Note: This is test shard 1 of 3.
[==========] Running 2 tests from 1 test suite.
[----------] Global test environment set-up.
[----------] 2 tests from CatalogManagerConfigurations/MasterStressTest
[ RUN ] CatalogManagerConfigurations/MasterStressTest.Test/0
WARNING: Logging before InitGoogleLogging() is written to STDERR
I20250905 19:52:10.448207 416 test_util.cc:276] Using random seed: 744883486
2025-09-05T19:52:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-09-05T19:52:10Z Disabled control of system clock
I20250905 19:52:10.482740 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38687
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
W20250905 19:52:10.821491 424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:10.821816 424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:10.821893 424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:10.829973 424 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 19:52:10.830088 424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:10.830150 424 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 19:52:10.830205 424 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 19:52:10.840415 424 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:10.842017 424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:10.843796 424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:52:10.853253 430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:10.853470 432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:10.853535 429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:10.854336 424 server_base.cc:1047] running on GCE node
I20250905 19:52:10.855122 424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:10.856593 424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:10.857786 424 hybrid_clock.cc:648] HybridClock initialized: now 1757101930857690 us; error 95 us; skew 500 ppm
I20250905 19:52:10.858109 424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:10.860431 424 webserver.cc:480] Webserver started at http://127.0.104.62:46537/ using document root <none> and password file <none>
I20250905 19:52:10.861191 424 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:10.861284 424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:10.861589 424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:52:10.864048 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/instance:
uuid: "c401c9f3020b4ae5931c566c6bcb1711"
format_stamp: "Formatted at 2025-09-05 19:52:10 on dist-test-slave-b1r7"
I20250905 19:52:10.865288 424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal/instance:
uuid: "c401c9f3020b4ae5931c566c6bcb1711"
format_stamp: "Formatted at 2025-09-05 19:52:10 on dist-test-slave-b1r7"
I20250905 19:52:10.870779 424 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20250905 19:52:10.874202 438 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:10.875561 424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:10.875727 424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
uuid: "c401c9f3020b4ae5931c566c6bcb1711"
format_stamp: "Formatted at 2025-09-05 19:52:10 on dist-test-slave-b1r7"
I20250905 19:52:10.875932 424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:10.894032 424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:10.895435 424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:10.895749 424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:10.917483 424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38687
I20250905 19:52:10.917477 489 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38687 every 8 connection(s)
I20250905 19:52:10.919056 424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/info.pb
I20250905 19:52:10.920920 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 424
I20250905 19:52:10.921182 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal/instance
I20250905 19:52:10.923642 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:41317
--webserver_interface=127.0.104.61
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 19:52:10.923991 490 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:10.929860 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:52:10.946763 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
W20250905 19:52:10.947779 440 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:41317: connect: Connection refused (error 111)
W20250905 19:52:10.949272 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:41317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:41317: connect: Connection refused (error 111)
I20250905 19:52:10.998642 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } attempt: 1
W20250905 19:52:11.000769 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:41317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:41317: connect: Connection refused (error 111)
I20250905 19:52:11.069025 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } attempt: 2
W20250905 19:52:11.070941 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:41317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:41317: connect: Connection refused (error 111)
I20250905 19:52:11.162221 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } attempt: 3
W20250905 19:52:11.164057 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:41317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:41317: connect: Connection refused (error 111)
W20250905 19:52:11.267050 492 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:11.267390 492 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:11.267513 492 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:11.276194 492 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 19:52:11.276300 492 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:11.276397 492 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 19:52:11.276469 492 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 19:52:11.286696 492 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:41317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:11.288352 492 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:11.290241 492 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:52:11.298483 500 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:11.298434 501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:11.299130 503 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:11.300410 492 server_base.cc:1047] running on GCE node
I20250905 19:52:11.301204 492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:11.302642 492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:11.303859 492 hybrid_clock.cc:648] HybridClock initialized: now 1757101931303812 us; error 47 us; skew 500 ppm
I20250905 19:52:11.304237 492 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:11.306409 492 webserver.cc:480] Webserver started at http://127.0.104.61:43453/ using document root <none> and password file <none>
I20250905 19:52:11.307142 492 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:11.307247 492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:11.307345 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } attempt: 4
I20250905 19:52:11.307689 492 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
W20250905 19:52:11.309123 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.61:41317: Network error: Client connection negotiation failed: client connection to 127.0.104.61:41317: connect: Connection refused (error 111)
I20250905 19:52:11.310323 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data/instance:
uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:11.311085 492 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal/instance:
uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:11.316306 492 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20250905 19:52:11.319602 509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:11.320995 492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20250905 19:52:11.321156 492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal
uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:11.321321 492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:11.336998 492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:11.337859 492 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:11.338092 492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:11.359469 492 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:41317
I20250905 19:52:11.359481 560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:41317 every 8 connection(s)
I20250905 19:52:11.361054 492 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/data/info.pb
I20250905 19:52:11.365481 561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:11.371061 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 492
I20250905 19:52:11.371260 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-1/wal/instance
I20250905 19:52:11.371628 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:52:11.373207 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:34395
--webserver_interface=127.0.104.60
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 19:52:11.387753 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:52:11.397105 561 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } has no permanent_uuid. Determining permanent_uuid...
W20250905 19:52:11.398203 512 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
W20250905 19:52:11.399744 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
I20250905 19:52:11.449091 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 1
W20250905 19:52:11.450959 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
I20250905 19:52:11.519209 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 2
W20250905 19:52:11.520980 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
I20250905 19:52:11.608388 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } attempt: 5
I20250905 19:52:11.612248 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 3
W20250905 19:52:11.614765 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
I20250905 19:52:11.614974 490 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } has no permanent_uuid. Determining permanent_uuid...
W20250905 19:52:11.616806 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
I20250905 19:52:11.668087 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 1
W20250905 19:52:11.670047 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
W20250905 19:52:11.710732 564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:11.711548 564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:11.711655 564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:11.719766 564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 19:52:11.719871 564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:11.719934 564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 19:52:11.720042 564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 19:52:11.730170 564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:11.731779 564 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:11.733541 564 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:52:11.738301 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 2
W20250905 19:52:11.740669 490 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
W20250905 19:52:11.741950 572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:11.742650 571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:11.744195 564 server_base.cc:1047] running on GCE node
W20250905 19:52:11.744211 574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:11.745026 564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:11.746483 564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:11.747689 564 hybrid_clock.cc:648] HybridClock initialized: now 1757101931747647 us; error 33 us; skew 500 ppm
I20250905 19:52:11.748091 564 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:11.750260 564 webserver.cc:480] Webserver started at http://127.0.104.60:44671/ using document root <none> and password file <none>
I20250905 19:52:11.751048 564 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:11.751163 564 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:11.751511 564 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:52:11.753955 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/instance:
uuid: "e38333af29db4bf6af41ee5a25d6701e"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:11.754755 564 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal/instance:
uuid: "e38333af29db4bf6af41ee5a25d6701e"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:11.758019 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 4
W20250905 19:52:11.759814 561 consensus_peers.cc:648] Error getting permanent uuid from config peer 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111)
I20250905 19:52:11.760682 564 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20250905 19:52:11.764084 580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:11.765489 564 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20250905 19:52:11.765658 564 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
uuid: "e38333af29db4bf6af41ee5a25d6701e"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:11.765861 564 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:11.803313 564 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:11.804288 564 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:11.804556 564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:11.832172 564 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:34395
I20250905 19:52:11.832206 631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:34395 every 8 connection(s)
I20250905 19:52:11.834936 564 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
I20250905 19:52:11.835695 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 564
I20250905 19:52:11.835927 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal/instance
I20250905 19:52:11.840929 632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:11.847388 490 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 3
I20250905 19:52:11.850358 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:52:11.879817 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Bootstrap starting.
I20250905 19:52:11.879954 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:52:11.883872 490 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:11.885350 490 log.cc:826] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:11.887230 632 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } has no permanent_uuid. Determining permanent_uuid...
I20250905 19:52:11.888347 490 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: No bootstrap required, opened a new log
I20250905 19:52:11.893872 490 raft_consensus.cc:357] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:11.894181 490 raft_consensus.cc:383] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:11.894279 490 raft_consensus.cc:738] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c401c9f3020b4ae5931c566c6bcb1711, State: Initialized, Role: FOLLOWER
I20250905 19:52:11.894886 490 consensus_queue.cc:260] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:11.895838 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:11.896246 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:11.897495 490 sys_catalog.cc:564] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:52:11.898017 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Bootstrap starting.
W20250905 19:52:11.898361 444 tablet.cc:2378] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:11.902738 632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:11.903843 632 log.cc:826] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:11.907130 632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: No bootstrap required, opened a new log
W20250905 19:52:11.911041 651 catalog_manager.cc:1560] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 19:52:11.911173 651 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 19:52:11.914676 632 raft_consensus.cc:357] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:11.915066 632 raft_consensus.cc:383] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:11.915174 632 raft_consensus.cc:738] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38333af29db4bf6af41ee5a25d6701e, State: Initialized, Role: FOLLOWER
I20250905 19:52:11.908865 424 master_runner.cc:186] Error getting master registration for 127.0.104.61:41317: OK, instance_id { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" instance_seqno: 1757101931354169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250905 19:52:11.916124 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 19:52:11.915925 632 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:11.917062 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:11.917402 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:11.918151 632 sys_catalog.cc:564] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: configured and running, proceeding with master startup.
W20250905 19:52:11.941870 665 catalog_manager.cc:1560] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20250905 19:52:11.942133 665 catalog_manager.cc:875] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20250905 19:52:11.937692 564 master_runner.cc:186] Error getting master registration for 127.0.104.61:41317: OK, instance_id { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" instance_seqno: 1757101931354169 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20250905 19:52:11.944703 564 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 19:52:12.057153 586 tablet.cc:2378] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:12.059024 561 consensus_peers.cc:658] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } attempt: 5
I20250905 19:52:12.073786 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: Bootstrap starting.
I20250905 19:52:12.078099 561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:12.079536 561 log.cc:826] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:12.082860 561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: No bootstrap required, opened a new log
I20250905 19:52:12.083572 654 raft_consensus.cc:491] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:12.083847 654 raft_consensus.cc:513] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:12.085593 654 leader_election.cc:290] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317)
I20250905 19:52:12.086946 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:12.087441 465 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38333af29db4bf6af41ee5a25d6701e in term 0.
I20250905 19:52:12.090147 582 leader_election.cc:304] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e; no voters:
I20250905 19:52:12.090720 654 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:12.090864 654 raft_consensus.cc:491] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:12.090955 654 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:12.092708 561 raft_consensus.cc:357] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:12.093055 561 raft_consensus.cc:383] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:12.093158 561 raft_consensus.cc:738] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6c0f2b9d8cd457a9c3448034c3d5c91, State: Initialized, Role: FOLLOWER
I20250905 19:52:12.093916 561 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:12.096489 654 raft_consensus.cc:513] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:12.098191 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711"
I20250905 19:52:12.098433 465 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:12.102463 465 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38333af29db4bf6af41ee5a25d6701e in term 1.
I20250905 19:52:12.103188 582 leader_election.cc:304] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e; no voters:
I20250905 19:52:12.103904 669 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:12.104786 654 leader_election.cc:290] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 1 election: Requested vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317)
I20250905 19:52:12.106151 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.106822 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:12.107992 561 sys_catalog.cc:564] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:52:12.108101 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
I20250905 19:52:12.108595 535 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:12.111922 535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38333af29db4bf6af41ee5a25d6701e in term 1.
I20250905 19:52:12.113095 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" is_pre_election: true
I20250905 19:52:12.113364 536 raft_consensus.cc:2374] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate e38333af29db4bf6af41ee5a25d6701e in term 1. Re-sending same reply.
I20250905 19:52:12.113409 669 raft_consensus.cc:695] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 1 LEADER]: Becoming Leader. State: Replica: e38333af29db4bf6af41ee5a25d6701e, State: Running, Role: LEADER
I20250905 19:52:12.114015 669 consensus_queue.cc:237] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:12.123836 654 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38333af29db4bf6af41ee5a25d6701e. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.124161 654 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:12.134622 675 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:52:12.137609 675 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:52:12.151958 535 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Refusing update from remote peer e38333af29db4bf6af41ee5a25d6701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:12.153175 654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:12.161098 465 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Refusing update from remote peer e38333af29db4bf6af41ee5a25d6701e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:12.162263 654 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:12.194909 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38333af29db4bf6af41ee5a25d6701e. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.195217 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:12.202160 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38333af29db4bf6af41ee5a25d6701e. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.202458 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:12.205905 683 mvcc.cc:204] Tried to move back new op lower bound from 7197089514083536896 to 7197089513944875008. Current Snapshot: MvccSnapshot[applied={T|T < 7197089514083536896}]
I20250905 19:52:12.221801 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.222169 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:12.222496 669 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.222750 669 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:12.226768 640 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.226984 640 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:12.233366 668 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:12.233675 668 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:12.235141 675 catalog_manager.cc:1349] Generated new cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:12.235303 675 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:52:12.284734 675 catalog_manager.cc:1372] Generated new certificate authority record
I20250905 19:52:12.287111 675 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:52:12.302695 689 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:12.302819 689 catalog_manager.cc:1554] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: loading cluster ID for follower catalog manager: success
I20250905 19:52:12.306607 689 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: acquiring CA information for follower catalog manager: success
W20250905 19:52:12.307983 689 catalog_manager.cc:875] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20250905 19:52:12.310359 675 catalog_manager.cc:5955] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Generated new TSK 0
I20250905 19:52:12.311661 675 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:52:12.341967 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
W20250905 19:52:12.801754 693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:12.802052 693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:12.802173 693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:12.810997 693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:12.811235 693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20250905 19:52:12.813621 697 raft_consensus.cc:491] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:12.813870 697 raft_consensus.cc:513] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:12.815400 697 leader_election.cc:290] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:12.816058 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" is_pre_election: true
I20250905 19:52:12.816367 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:12.817178 441 leader_election.cc:304] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711; no voters: c6c0f2b9d8cd457a9c3448034c3d5c91, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:12.817591 697 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:12.823061 693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:12.824770 693 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:12.826700 693 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:52:12.837097 700 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:12.837160 699 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:12.839032 702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:12.839641 693 server_base.cc:1047] running on GCE node
I20250905 19:52:12.840391 693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:12.841594 693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:12.842828 693 hybrid_clock.cc:648] HybridClock initialized: now 1757101932842769 us; error 56 us; skew 500 ppm
I20250905 19:52:12.843220 693 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:12.846006 693 webserver.cc:480] Webserver started at http://127.0.104.1:44193/ using document root <none> and password file <none>
I20250905 19:52:12.846946 693 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:12.847077 693 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:12.847481 693 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:52:12.850934 693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data/instance:
uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
format_stamp: "Formatted at 2025-09-05 19:52:12 on dist-test-slave-b1r7"
I20250905 19:52:12.851872 693 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/instance:
uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
format_stamp: "Formatted at 2025-09-05 19:52:12 on dist-test-slave-b1r7"
I20250905 19:52:12.858865 693 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20250905 19:52:12.862462 708 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:12.864024 693 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:12.864205 693 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal
uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
format_stamp: "Formatted at 2025-09-05 19:52:12 on dist-test-slave-b1r7"
I20250905 19:52:12.864419 693 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:12.894809 693 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:12.895737 693 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:12.896063 693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:12.897073 693 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:52:12.898838 693 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:52:12.898945 693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:12.899080 693 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:52:12.899153 693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:12.913352 651 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:12.913501 651 catalog_manager.cc:1554] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: loading cluster ID for follower catalog manager: success
I20250905 19:52:12.917649 651 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: acquiring CA information for follower catalog manager: success
I20250905 19:52:12.920168 651 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:52:12.942680 693 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34531
I20250905 19:52:12.942781 820 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34531 every 8 connection(s)
I20250905 19:52:12.944531 693 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/data/info.pb
I20250905 19:52:12.946684 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 693
I20250905 19:52:12.946899 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/instance
I20250905 19:52:12.952659 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250905 19:52:12.969884 821 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:12.969918 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:12.970384 822 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:12.971575 822 heartbeater.cc:507] Master 127.0.104.62:38687 requested a full tablet report, sending...
I20250905 19:52:12.970386 821 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:12.972661 821 heartbeater.cc:507] Master 127.0.104.60:34395 requested a full tablet report, sending...
I20250905 19:52:12.974287 454 ts_manager.cc:194] Registered new tserver with Master: f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:12.974395 597 ts_manager.cc:194] Registered new tserver with Master: f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:12.976800 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:55787
I20250905 19:52:12.977416 823 heartbeater.cc:344] Connected to a master server at 127.0.104.61:41317
I20250905 19:52:12.977562 823 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:12.978137 823 heartbeater.cc:507] Master 127.0.104.61:41317 requested a full tablet report, sending...
I20250905 19:52:12.981355 526 ts_manager.cc:194] Registered new tserver with Master: f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:13.250993 832 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:13.251206 832 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:13.252540 832 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:13.253144 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:13.253124 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:13.253782 511 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:13.254117 832 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:13.310667 689 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20250905 19:52:13.321600 830 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:13.321965 830 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:13.322098 830 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:13.331044 830 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:13.331302 830 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20250905 19:52:13.341106 830 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:13.342784 830 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:13.344898 830 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:52:13.355476 837 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:13.355548 838 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:13.356781 840 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:13.357415 830 server_base.cc:1047] running on GCE node
I20250905 19:52:13.358042 830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:13.362071 830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:13.363307 830 hybrid_clock.cc:648] HybridClock initialized: now 1757101933363240 us; error 68 us; skew 500 ppm
I20250905 19:52:13.363732 830 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:13.366459 830 webserver.cc:480] Webserver started at http://127.0.104.2:40363/ using document root <none> and password file <none>
I20250905 19:52:13.367442 830 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:13.367558 830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:13.367944 830 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:52:13.371388 830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data/instance:
uuid: "835daf30e4324782848290be181ae78f"
format_stamp: "Formatted at 2025-09-05 19:52:13 on dist-test-slave-b1r7"
I20250905 19:52:13.372344 830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/instance:
uuid: "835daf30e4324782848290be181ae78f"
format_stamp: "Formatted at 2025-09-05 19:52:13 on dist-test-slave-b1r7"
I20250905 19:52:13.378206 830 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20250905 19:52:13.382222 846 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:13.383571 830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20250905 19:52:13.383713 830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal
uuid: "835daf30e4324782848290be181ae78f"
format_stamp: "Formatted at 2025-09-05 19:52:13 on dist-test-slave-b1r7"
I20250905 19:52:13.383884 830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:13.402822 830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:13.403642 830 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:13.403883 830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:13.404889 830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:52:13.406944 830 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:52:13.407039 830 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:13.407116 830 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:52:13.407173 830 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:13.441785 830 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35993
I20250905 19:52:13.441838 958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35993 every 8 connection(s)
I20250905 19:52:13.443204 830 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/data/info.pb
I20250905 19:52:13.445883 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 830
I20250905 19:52:13.446102 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/instance
I20250905 19:52:13.450775 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--heartbeat_rpc_timeout_ms=1000
--heartbeat_max_failures_before_backoff=1
--heartbeat_interval_ms=500 with env {}
I20250905 19:52:13.466339 959 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:13.466508 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:13.466549 961 heartbeater.cc:344] Connected to a master server at 127.0.104.61:41317
I20250905 19:52:13.466806 960 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:13.466809 961 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:13.466812 959 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:13.468298 961 heartbeater.cc:507] Master 127.0.104.61:41317 requested a full tablet report, sending...
I20250905 19:52:13.468338 960 heartbeater.cc:507] Master 127.0.104.62:38687 requested a full tablet report, sending...
I20250905 19:52:13.468465 959 heartbeater.cc:507] Master 127.0.104.60:34395 requested a full tablet report, sending...
I20250905 19:52:13.470309 597 ts_manager.cc:194] Registered new tserver with Master: 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:13.470364 454 ts_manager.cc:194] Registered new tserver with Master: 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:13.471051 526 ts_manager.cc:194] Registered new tserver with Master: 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:13.471588 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:54949
I20250905 19:52:13.480150 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:13.803440 966 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:13.804239 966 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:13.804329 966 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:13.812412 966 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:13.812594 966 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20250905 19:52:13.822901 966 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--raft_heartbeat_interval_ms=1000
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--heartbeat_interval_ms=500
--heartbeat_max_failures_before_backoff=1
--heartbeat_rpc_timeout_ms=1000
--tserver_master_addrs=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:13.824795 966 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:13.826984 966 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20250905 19:52:13.836658 971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:13.840866 972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:13.841779 974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:13.842300 966 server_base.cc:1047] running on GCE node
I20250905 19:52:13.842864 966 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:13.843783 966 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:13.845069 966 hybrid_clock.cc:648] HybridClock initialized: now 1757101933845024 us; error 66 us; skew 500 ppm
I20250905 19:52:13.845381 966 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:13.847568 966 webserver.cc:480] Webserver started at http://127.0.104.3:40373/ using document root <none> and password file <none>
I20250905 19:52:13.848404 966 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:13.848513 966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:13.848968 966 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20250905 19:52:13.849032 979 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:13.849220 979 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:13.849807 979 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:13.850350 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:13.850358 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:13.851059 512 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:13.851338 979 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:13.851945 966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data/instance:
uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
format_stamp: "Formatted at 2025-09-05 19:52:13 on dist-test-slave-b1r7"
I20250905 19:52:13.852711 966 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/instance:
uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
format_stamp: "Formatted at 2025-09-05 19:52:13 on dist-test-slave-b1r7"
I20250905 19:52:13.857923 966 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20250905 19:52:13.861850 981 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:13.863188 966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:13.863345 966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal
uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
format_stamp: "Formatted at 2025-09-05 19:52:13 on dist-test-slave-b1r7"
I20250905 19:52:13.863541 966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:13.865494 982 raft_consensus.cc:491] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:13.865692 982 raft_consensus.cc:513] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:13.866407 982 leader_election.cc:290] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:13.866897 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" is_pre_election: true
I20250905 19:52:13.866902 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:13.867193 535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c401c9f3020b4ae5931c566c6bcb1711 in term 1.
I20250905 19:52:13.867666 440 leader_election.cc:304] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711, c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:13.867944 982 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:13.868089 982 raft_consensus.cc:491] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Starting leader election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:13.868189 982 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:13.870229 982 raft_consensus.cc:513] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:13.870955 982 leader_election.cc:290] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 2 election: Requested vote from peers c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:13.871579 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
I20250905 19:52:13.871594 607 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e"
I20250905 19:52:13.871793 535 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:13.874382 535 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c401c9f3020b4ae5931c566c6bcb1711 in term 2.
I20250905 19:52:13.874801 440 leader_election.cc:304] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711, c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:13.875083 982 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:13.875478 982 raft_consensus.cc:695] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 LEADER]: Becoming Leader. State: Replica: c401c9f3020b4ae5931c566c6bcb1711, State: Running, Role: LEADER
I20250905 19:52:13.875895 982 consensus_queue.cc:237] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:13.877590 987 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c401c9f3020b4ae5931c566c6bcb1711. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:13.877820 987 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:13.878405 989 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:52:13.880553 989 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:52:13.881517 989 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:13.881600 989 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:52:13.882882 989 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:52:13.883785 989 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Loaded TSK: 0
I20250905 19:52:13.885362 989 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:52:13.894018 966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:13.894833 966 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:13.895077 966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:13.896052 966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20250905 19:52:13.897656 966 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20250905 19:52:13.897749 966 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:13.897826 966 ts_tablet_manager.cc:610] Registered 0 tablets
I20250905 19:52:13.897881 966 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:13.944119 966 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:43385
I20250905 19:52:13.944185 1097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:43385 every 8 connection(s)
I20250905 19:52:13.945590 966 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/data/info.pb
I20250905 19:52:13.953300 464 raft_consensus.cc:1238] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 LEADER]: Rejecting Update request from peer e38333af29db4bf6af41ee5a25d6701e for earlier term 1. Current term is 2. Ops: []
I20250905 19:52:13.953955 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 966
I20250905 19:52:13.954154 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/instance
I20250905 19:52:13.954082 1105 consensus_queue.cc:1046] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 }, Status: INVALID_TERM, Last received: 1.4, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:52:13.973757 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:13.977162 1098 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:13.977706 1098 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:13.979315 1098 heartbeater.cc:507] Master 127.0.104.60:34395 requested a full tablet report, sending...
I20250905 19:52:13.981410 597 ts_manager.cc:194] Registered new tserver with Master: 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:13.982072 1105 consensus_queue.cc:1230] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Peer c401c9f3020b4ae5931c566c6bcb1711 log is divergent from this leader: its last log entry 2.5 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 4
I20250905 19:52:13.982506 597 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:58249
I20250905 19:52:13.982583 1107 raft_consensus.cc:3053] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:52:13.982731 1107 raft_consensus.cc:738] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 1 LEADER]: Becoming Follower/Learner. State: Replica: e38333af29db4bf6af41ee5a25d6701e, State: Running, Role: LEADER
I20250905 19:52:13.983152 1107 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:13.983466 1107 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:13.983865 535 raft_consensus.cc:1238] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Rejecting Update request from peer e38333af29db4bf6af41ee5a25d6701e for earlier term 1. Current term is 2. Ops: []
I20250905 19:52:13.984524 1100 heartbeater.cc:344] Connected to a master server at 127.0.104.61:41317
I20250905 19:52:13.984675 1100 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:13.985038 1100 heartbeater.cc:507] Master 127.0.104.61:41317 requested a full tablet report, sending...
I20250905 19:52:13.986177 525 ts_manager.cc:194] Registered new tserver with Master: 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:13.988286 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:13.988421 1099 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:13.988783 1099 heartbeater.cc:507] Master 127.0.104.62:38687 requested a full tablet report, sending...
I20250905 19:52:13.989619 454 ts_manager.cc:194] Registered new tserver with Master: 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:13.991832 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20250905 19:52:14.023789 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_050ef321cf1047de9e72247fd6ba1f08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.023912 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_cf5aee17e0e64dfead8027a21332f5ed"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.023746 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_41880d70e404433eba991433a87d8964"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.024134 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_cf9ab68443424d3dbf3d5a7ee03a560b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.025689 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41880d70e404433eba991433a87d8964 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.025705 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf5aee17e0e64dfead8027a21332f5ed in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.027058 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cf9ab68443424d3dbf3d5a7ee03a560b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.028946 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_050ef321cf1047de9e72247fd6ba1f08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.025738 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_d024b68e35f94c8eaf705ccb4d70186b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.032087 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d024b68e35f94c8eaf705ccb4d70186b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.032011 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8839f23e38974ec983b330ab7eb76511"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.033212 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c10b19058d2043e79c1aa1ca19de6dfc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.037658 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8839f23e38974ec983b330ab7eb76511 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.031611 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6de2c91156394d2ab216f44208a340a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.036514 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c430e1fa0a5b4a7399463e5c4c76f741"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.038965 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c10b19058d2043e79c1aa1ca19de6dfc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.039211 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c430e1fa0a5b4a7399463e5c4c76f741 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.040134 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6de2c91156394d2ab216f44208a340a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.030735 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_aa485619b8bd4b36b77da7474d2ba55d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.040829 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa485619b8bd4b36b77da7474d2ba55d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.042528 535 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Refusing update from remote peer c401c9f3020b4ae5931c566c6bcb1711: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250905 19:52:14.042723 607 raft_consensus.cc:1273] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Refusing update from remote peer c401c9f3020b4ae5931c566c6bcb1711: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20250905 19:52:14.043165 987 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:52:14.043495 982 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20250905 19:52:14.061172 979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c401c9f3020b4ae5931c566c6bcb1711. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:14.061486 979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:14.062424 1107 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c401c9f3020b4ae5931c566c6bcb1711. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:14.062718 1107 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:14.076107 1135 mvcc.cc:204] Tried to move back new op lower bound from 7197089521836871680 to 7197089521159528448. Current Snapshot: MvccSnapshot[applied={T|T < 7197089521836871680 or (T in {7197089521836871680})}]
I20250905 19:52:14.087716 1107 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:14.088009 1107 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:14.112954 979 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:14.113206 979 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:14.120173 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu pid 564
I20250905 19:52:14.121484 627 generic_service.cc:196] Checking for leaks (request via RPC)
I20250905 19:52:14.122059 982 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:14.122303 982 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:14.131431 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d024b68e35f94c8eaf705ccb4d70186b" } new_table_name: "default.Table_756245ad032543e783c35761fd327d32" modify_external_catalogs: true
I20250905 19:52:14.132061 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c10b19058d2043e79c1aa1ca19de6dfc" } modify_external_catalogs: true
I20250905 19:52:14.132321 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c10b19058d2043e79c1aa1ca19de6dfc" } modify_external_catalogs: true
I20250905 19:52:14.135185 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6de2c91156394d2ab216f44208a340a5" } modify_external_catalogs: true
I20250905 19:52:14.135342 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_050ef321cf1047de9e72247fd6ba1f08" } modify_external_catalogs: true
I20250905 19:52:14.135442 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6de2c91156394d2ab216f44208a340a5" } modify_external_catalogs: true
I20250905 19:52:14.135550 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_050ef321cf1047de9e72247fd6ba1f08" } modify_external_catalogs: true
I20250905 19:52:14.136597 987 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:14.136956 987 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:14.137208 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_aa485619b8bd4b36b77da7474d2ba55d" } modify_external_catalogs: true
I20250905 19:52:14.137389 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_aa485619b8bd4b36b77da7474d2ba55d" } modify_external_catalogs: true
I20250905 19:52:14.136967 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_cf9ab68443424d3dbf3d5a7ee03a560b" } new_table_name: "default.Table_5b9c5afac54648e19d47aca2d54aa228" modify_external_catalogs: true
I20250905 19:52:14.139393 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8839f23e38974ec983b330ab7eb76511" } new_table_name: "default.Table_fc44097d961e4e8ab5ce95984b17bc16" modify_external_catalogs: true
I20250905 19:52:14.142211 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c430e1fa0a5b4a7399463e5c4c76f741" } modify_external_catalogs: true
I20250905 19:52:14.142535 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_41880d70e404433eba991433a87d8964" } new_table_name: "default.Table_01c28a327aad4e16bd12a580ba6ce0d4" modify_external_catalogs: true
I20250905 19:52:14.142246 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_cf5aee17e0e64dfead8027a21332f5ed" } new_table_name: "default.Table_02fc89d2523848a4aba495c72a35491f" modify_external_catalogs: true
I20250905 19:52:14.142463 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c430e1fa0a5b4a7399463e5c4c76f741" } modify_external_catalogs: true
W20250905 19:52:14.148597 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92ea9fe1c36c4807ba02f23e6877198c has no consensus state
W20250905 19:52:14.150342 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1f668b9ae964b7ebf87726f734e3a40 has no consensus state
I20250905 19:52:14.155576 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 9131405eeee443a484998f988c49e95f
W20250905 19:52:14.156338 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c7c015508fe7465d852fe3875a8b1aa5 has no consensus state
I20250905 19:52:14.156885 451 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 59522fbf4a8341bab20ec85569278ab6
W20250905 19:52:14.161468 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 417199577c6041b5801756876da3448a has no consensus state
I20250905 19:52:14.161768 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 5d63649114c44dc89d60d0cb73adae42
W20250905 19:52:14.163250 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4d21be9d7e5140b6aae49e12f0509b12 has no consensus state
I20250905 19:52:14.163614 455 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c6a1260657d246b88f2560690bb75efb
I20250905 19:52:14.167716 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 4531122a388b4018a7087e921a8ce6cb
W20250905 19:52:14.190224 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d1f668b9ae964b7ebf87726f734e3a40 consensus state has no leader
I20250905 19:52:14.200121 891 tablet_service.cc:1468] Processing CreateTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 (DEFAULT_TABLE table=default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.201164 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f668b9ae964b7ebf87726f734e3a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.202365 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c (DEFAULT_TABLE table=default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.203018 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 4d21be9d7e5140b6aae49e12f0509b12 (DEFAULT_TABLE table=default.Table_01c28a327aad4e16bd12a580ba6ce0d4 [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.203603 1029 tablet_service.cc:1468] Processing CreateTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 (DEFAULT_TABLE table=default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.204074 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ea9fe1c36c4807ba02f23e6877198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.203884 756 tablet_service.cc:1468] Processing CreateTablet for tablet 417199577c6041b5801756876da3448a (DEFAULT_TABLE table=default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.204411 754 tablet_service.cc:1468] Processing CreateTablet for tablet 4d21be9d7e5140b6aae49e12f0509b12 (DEFAULT_TABLE table=default.Table_01c28a327aad4e16bd12a580ba6ce0d4 [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.204743 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 (DEFAULT_TABLE table=default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.204736 892 tablet_service.cc:1468] Processing CreateTablet for tablet 4d21be9d7e5140b6aae49e12f0509b12 (DEFAULT_TABLE table=default.Table_01c28a327aad4e16bd12a580ba6ce0d4 [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.205178 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d21be9d7e5140b6aae49e12f0509b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.205194 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f668b9ae964b7ebf87726f734e3a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.202344 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 417199577c6041b5801756876da3448a (DEFAULT_TABLE table=default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.205775 894 tablet_service.cc:1468] Processing CreateTablet for tablet 417199577c6041b5801756876da3448a (DEFAULT_TABLE table=default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.205960 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d21be9d7e5140b6aae49e12f0509b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.206203 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417199577c6041b5801756876da3448a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.206207 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417199577c6041b5801756876da3448a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.206486 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c015508fe7465d852fe3875a8b1aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.206739 893 tablet_service.cc:1468] Processing CreateTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c (DEFAULT_TABLE table=default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.207135 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ea9fe1c36c4807ba02f23e6877198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.199373 890 tablet_service.cc:1468] Processing CreateTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 (DEFAULT_TABLE table=default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.207593 752 tablet_service.cc:1468] Processing CreateTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 (DEFAULT_TABLE table=default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.207938 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c015508fe7465d852fe3875a8b1aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.207952 755 tablet_service.cc:1468] Processing CreateTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c (DEFAULT_TABLE table=default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.208209 753 tablet_service.cc:1468] Processing CreateTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 (DEFAULT_TABLE table=default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.223086 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 417199577c6041b5801756876da3448a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.223644 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1f668b9ae964b7ebf87726f734e3a40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.226419 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d21be9d7e5140b6aae49e12f0509b12. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.228991 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92ea9fe1c36c4807ba02f23e6877198c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.230661 1169 tablet_bootstrap.cc:492] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.240255 1169 tablet_bootstrap.cc:654] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.240418 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7c015508fe7465d852fe3875a8b1aa5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.242018 1169 log.cc:826] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:14.243903 1171 tablet_bootstrap.cc:492] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.244930 1170 tablet_bootstrap.cc:492] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.248842 1171 tablet_bootstrap.cc:654] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.249702 1170 tablet_bootstrap.cc:654] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.251413 1170 log.cc:826] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:14.257591 1169 tablet_bootstrap.cc:492] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.257906 1170 tablet_bootstrap.cc:492] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.257899 1169 ts_tablet_manager.cc:1397] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.006s sys 0.005s
I20250905 19:52:14.258297 1170 ts_tablet_manager.cc:1397] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.005s
I20250905 19:52:14.264101 1171 log.cc:826] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:14.266595 1170 raft_consensus.cc:357] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.266817 1169 raft_consensus.cc:357] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.267120 1170 raft_consensus.cc:383] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.267143 1169 raft_consensus.cc:383] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.267257 1169 raft_consensus.cc:738] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.267256 1170 raft_consensus.cc:738] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.267935 1169 consensus_queue.cc:260] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.267939 1170 consensus_queue.cc:260] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.268945 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:14.270666 1170 ts_tablet_manager.cc:1428] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20250905 19:52:14.271028 822 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:14.272193 1169 ts_tablet_manager.cc:1428] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.010s sys 0.000s
I20250905 19:52:14.272543 1169 tablet_bootstrap.cc:492] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.268939 1099 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:14.274444 1169 tablet_bootstrap.cc:654] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.274384 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet c7c015508fe7465d852fe3875a8b1aa5 (table default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 1 got 0
I20250905 19:52:14.274668 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 1 got 0
I20250905 19:52:14.274856 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]). Expected version 1 got 0
I20250905 19:52:14.275049 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 417199577c6041b5801756876da3448a (table default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]). Expected version 1 got 0
I20250905 19:52:14.275192 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 4d21be9d7e5140b6aae49e12f0509b12 (table default.Table_01c28a327aad4e16bd12a580ba6ce0d4 [id=7a9f1417bfd04c158dbedc9ea02d6dce]). Expected version 1 got 0
I20250905 19:52:14.277235 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet c7c015508fe7465d852fe3875a8b1aa5 (table default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 1 got 0
I20250905 19:52:14.277436 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 4d21be9d7e5140b6aae49e12f0509b12 (table default.Table_01c28a327aad4e16bd12a580ba6ce0d4 [id=7a9f1417bfd04c158dbedc9ea02d6dce]). Expected version 1 got 0
I20250905 19:52:14.277559 1171 tablet_bootstrap.cc:492] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.277602 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 417199577c6041b5801756876da3448a (table default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]). Expected version 1 got 0
I20250905 19:52:14.277753 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 1 got 0
I20250905 19:52:14.277853 1171 ts_tablet_manager.cc:1397] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.034s user 0.011s sys 0.000s
I20250905 19:52:14.277912 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]). Expected version 1 got 0
I20250905 19:52:14.280251 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0a4b61de931546689e038111b0a405a4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.280826 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0a4b61de931546689e038111b0a405a4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.282052 1170 tablet_bootstrap.cc:492] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.283852 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e7d692a2d299432090b66b38168b5894"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.284030 1170 tablet_bootstrap.cc:654] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:14.284539 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e7d692a2d299432090b66b38168b5894 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.294834 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 4d21be9d7e5140b6aae49e12f0509b12 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.296190 1170 tablet_bootstrap.cc:492] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.296382 1170 ts_tablet_manager.cc:1397] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.024s user 0.009s sys 0.000s
I20250905 19:52:14.296759 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 4d21be9d7e5140b6aae49e12f0509b12 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.297063 454 master_service.cc:948] ReplaceTablet: received request to replace tablet c7c015508fe7465d852fe3875a8b1aa5 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.297312 451 master_service.cc:948] ReplaceTablet: received request to replace tablet c7c015508fe7465d852fe3875a8b1aa5 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.297552 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 4d21be9d7e5140b6aae49e12f0509b12 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.297442 1170 raft_consensus.cc:357] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.297662 1170 raft_consensus.cc:383] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.297744 1170 raft_consensus.cc:738] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.298036 1170 consensus_queue.cc:260] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.298599 1170 ts_tablet_manager.cc:1428] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:14.300333 1170 tablet_bootstrap.cc:492] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.302078 1170 tablet_bootstrap.cc:654] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.303913 1171 raft_consensus.cc:357] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.304471 1171 raft_consensus.cc:383] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.304610 1171 raft_consensus.cc:738] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.305902 1033 tablet_service.cc:1468] Processing CreateTablet for tablet ca53e56561674c8c830a0e28443a19e0 (DEFAULT_TABLE table=default.Table_0a4b61de931546689e038111b0a405a4 [id=b72b466b7abe468dbaa985c9f469cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.306044 1169 tablet_bootstrap.cc:492] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.306893 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca53e56561674c8c830a0e28443a19e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.306964 1169 ts_tablet_manager.cc:1397] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250905 19:52:14.308393 1169 raft_consensus.cc:357] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.308689 1169 raft_consensus.cc:383] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.308673 891 tablet_service.cc:1468] Processing CreateTablet for tablet ca53e56561674c8c830a0e28443a19e0 (DEFAULT_TABLE table=default.Table_0a4b61de931546689e038111b0a405a4 [id=b72b466b7abe468dbaa985c9f469cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.308775 1169 raft_consensus.cc:738] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.309054 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca53e56561674c8c830a0e28443a19e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.309594 1169 consensus_queue.cc:260] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.310271 1171 consensus_queue.cc:260] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.311192 1170 tablet_bootstrap.cc:492] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.311334 1170 ts_tablet_manager.cc:1397] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20250905 19:52:14.312206 1170 raft_consensus.cc:357] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.312381 1170 raft_consensus.cc:383] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.312469 1170 raft_consensus.cc:738] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.312652 1170 consensus_queue.cc:260] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.313409 1169 ts_tablet_manager.cc:1428] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 19:52:14.313725 1169 tablet_bootstrap.cc:492] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.315456 1169 tablet_bootstrap.cc:654] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.315412 756 tablet_service.cc:1468] Processing CreateTablet for tablet ca53e56561674c8c830a0e28443a19e0 (DEFAULT_TABLE table=default.Table_0a4b61de931546689e038111b0a405a4 [id=b72b466b7abe468dbaa985c9f469cbcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.315760 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca53e56561674c8c830a0e28443a19e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.316174 1170 ts_tablet_manager.cc:1428] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 19:52:14.316517 1170 tablet_bootstrap.cc:492] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.319375 1170 tablet_bootstrap.cc:654] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.321905 1170 tablet_bootstrap.cc:492] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.321931 1169 tablet_bootstrap.cc:492] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.322055 1169 ts_tablet_manager.cc:1397] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20250905 19:52:14.322084 1170 ts_tablet_manager.cc:1397] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.002s
I20250905 19:52:14.322800 1170 raft_consensus.cc:357] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.322824 1169 raft_consensus.cc:357] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.322988 1169 raft_consensus.cc:383] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.323010 1170 raft_consensus.cc:383] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.323072 1169 raft_consensus.cc:738] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.323096 1170 raft_consensus.cc:738] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.323308 1169 consensus_queue.cc:260] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.323380 1170 consensus_queue.cc:260] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.324087 1170 ts_tablet_manager.cc:1428] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:14.324378 1170 tablet_bootstrap.cc:492] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.324352 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet c7c015508fe7465d852fe3875a8b1aa5
I20250905 19:52:14.325088 454 catalog_manager.cc:6652] ReplaceTablet: tablet c7c015508fe7465d852fe3875a8b1aa5 deleted and replaced by tablet c2a054ea3bca416b8fa344a5de28d798
I20250905 19:52:14.325698 1171 ts_tablet_manager.cc:1428] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.048s user 0.011s sys 0.003s
I20250905 19:52:14.326051 1170 tablet_bootstrap.cc:654] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.328326 960 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:14.328971 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2a054ea3bca416b8fa344a5de28d798) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.330539 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2a054ea3bca416b8fa344a5de28d798) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.332218 754 tablet_service.cc:1515] Processing DeleteTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2a054ea3bca416b8fa344a5de28d798) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.332309 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_893d44ae3142460cbfbc43ac8756dd02"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.333107 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_893d44ae3142460cbfbc43ac8756dd02 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.335108 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1a37863c1a764e7eb95e92d19da52a61"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.335933 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1a37863c1a764e7eb95e92d19da52a61 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.339819 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_da585f1b2ba344b2a01227658054e1a1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.340471 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_da585f1b2ba344b2a01227658054e1a1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.342576 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ffb30e1b0503476dbad2d006c2f66661"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.343377 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ffb30e1b0503476dbad2d006c2f66661 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.344985 1171 tablet_bootstrap.cc:492] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.346930 1171 tablet_bootstrap.cc:654] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.347793 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4d21be9d7e5140b6aae49e12f0509b12
I20250905 19:52:14.348402 452 catalog_manager.cc:6652] ReplaceTablet: tablet 4d21be9d7e5140b6aae49e12f0509b12 deleted and replaced by tablet 987a6193d2a74f90906a45b237040e39
I20250905 19:52:14.348716 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_06206a701b6e4278a053ea096bff8659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.349267 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_06206a701b6e4278a053ea096bff8659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.350729 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_3db28a103a6f46f08d3aeaf8fb3432d5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.351191 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.351768 1169 ts_tablet_manager.cc:1428] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250905 19:52:14.352125 1169 tablet_bootstrap.cc:492] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.353209 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_accba7263cc04e558868883f599f2838"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.353750 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_accba7263cc04e558868883f599f2838 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.354451 1169 tablet_bootstrap.cc:654] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.355641 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d21be9d7e5140b6aae49e12f0509b12 with delete_type TABLET_DATA_DELETED (replaced by tablet 987a6193d2a74f90906a45b237040e39) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.356884 451 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]). Expected version 1 got 0
I20250905 19:52:14.357028 1171 tablet_bootstrap.cc:492] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.357095 451 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 1 got 0
I20250905 19:52:14.357251 1171 ts_tablet_manager.cc:1397] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250905 19:52:14.358042 451 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 417199577c6041b5801756876da3448a (table default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]). Expected version 1 got 0
I20250905 19:52:14.359647 1186 tablet_replica.cc:331] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:14.363579 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d21be9d7e5140b6aae49e12f0509b12 with delete_type TABLET_DATA_DELETED (replaced by tablet 987a6193d2a74f90906a45b237040e39) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.364307 1186 raft_consensus.cc:2241] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.364560 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0b23b83842434d578f1044e4ff7b37f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.364985 1186 raft_consensus.cc:2270] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:14.365131 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b23b83842434d578f1044e4ff7b37f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.366997 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet c7c015508fe7465d852fe3875a8b1aa5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c7c015508fe7465d852fe3875a8b1aa5 already in progress: creating tablet
I20250905 19:52:14.358878 1171 raft_consensus.cc:357] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.370471 1171 raft_consensus.cc:383] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.370577 1171 raft_consensus.cc:738] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.370990 1171 consensus_queue.cc:260] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.372092 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d21be9d7e5140b6aae49e12f0509b12 with delete_type TABLET_DATA_DELETED (replaced by tablet 987a6193d2a74f90906a45b237040e39) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.373960 1186 ts_tablet_manager.cc:1905] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.374284 1171 ts_tablet_manager.cc:1428] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.017s user 0.002s sys 0.001s
I20250905 19:52:14.378129 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_756245ad032543e783c35761fd327d32" } new_table_name: "default.Table_b265ba1d975f43fe9d26542d0296d10d" modify_external_catalogs: true
I20250905 19:52:14.378633 1185 tablet_replica.cc:331] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:14.381340 1169 tablet_bootstrap.cc:492] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.381528 1169 ts_tablet_manager.cc:1397] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.001s sys 0.003s
I20250905 19:52:14.382335 1169 raft_consensus.cc:357] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.382546 1169 raft_consensus.cc:383] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.382520 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fc44097d961e4e8ab5ce95984b17bc16" } new_table_name: "default.Table_01b46213458147cd9bf884185bcff837" modify_external_catalogs: true
I20250905 19:52:14.382625 1169 raft_consensus.cc:738] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.383440 1169 consensus_queue.cc:260] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.383251 1171 tablet_bootstrap.cc:492] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.386134 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_01c28a327aad4e16bd12a580ba6ce0d4" } new_table_name: "default.Table_d305bf057f3f4754bc3dab037bd823bd" modify_external_catalogs: true
I20250905 19:52:14.387214 1185 raft_consensus.cc:2241] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.387728 1185 raft_consensus.cc:2270] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.390650 1171 tablet_bootstrap.cc:654] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.392794 1169 ts_tablet_manager.cc:1428] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:14.393260 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0a4b61de931546689e038111b0a405a4" } modify_external_catalogs: true
I20250905 19:52:14.393442 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0a4b61de931546689e038111b0a405a4" } modify_external_catalogs: true
I20250905 19:52:14.393885 1185 ts_tablet_manager.cc:1905] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.397404 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_02fc89d2523848a4aba495c72a35491f" } modify_external_catalogs: true
I20250905 19:52:14.397563 1169 tablet_bootstrap.cc:492] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.397606 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_02fc89d2523848a4aba495c72a35491f" } modify_external_catalogs: true
I20250905 19:52:14.399266 1186 ts_tablet_manager.cc:1918] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.399577 1186 log.cc:1199] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/c7c015508fe7465d852fe3875a8b1aa5
I20250905 19:52:14.400187 1186 ts_tablet_manager.cc:1939] T c7c015508fe7465d852fe3875a8b1aa5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.400928 1169 tablet_bootstrap.cc:654] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.402953 448 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_756245ad032543e783c35761fd327d32 [id=95b523055663453a83fe91e35e1205cc]). Expected version 1 got 0
I20250905 19:52:14.403746 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 1 got 0
W20250905 19:52:14.404713 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 92ea9fe1c36c4807ba02f23e6877198c consensus state has no leader
I20250905 19:52:14.405079 893 tablet_service.cc:1468] Processing CreateTablet for tablet c2a054ea3bca416b8fa344a5de28d798 (DEFAULT_TABLE table=default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.405517 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a054ea3bca416b8fa344a5de28d798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.408886 1170 tablet_bootstrap.cc:492] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.409065 1170 ts_tablet_manager.cc:1397] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.085s user 0.019s sys 0.028s
I20250905 19:52:14.409384 1171 tablet_bootstrap.cc:492] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.409569 1171 ts_tablet_manager.cc:1397] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.001s
I20250905 19:52:14.409973 1170 raft_consensus.cc:357] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.410176 1170 raft_consensus.cc:383] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.410279 1170 raft_consensus.cc:738] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.410472 1171 raft_consensus.cc:357] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.410529 1170 consensus_queue.cc:260] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.410650 1171 raft_consensus.cc:383] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.410761 1171 raft_consensus.cc:738] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.410804 756 tablet_service.cc:1468] Processing CreateTablet for tablet c2a054ea3bca416b8fa344a5de28d798 (DEFAULT_TABLE table=default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.411255 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a054ea3bca416b8fa344a5de28d798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.411325 1171 consensus_queue.cc:260] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.412887 1171 ts_tablet_manager.cc:1428] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
W20250905 19:52:14.414049 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c2a054ea3bca416b8fa344a5de28d798 consensus state has no leader
W20250905 19:52:14.414587 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 987a6193d2a74f90906a45b237040e39 has no consensus state
I20250905 19:52:14.414975 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ffb30e1b0503476dbad2d006c2f66661" } modify_external_catalogs: true
I20250905 19:52:14.415184 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ffb30e1b0503476dbad2d006c2f66661" } modify_external_catalogs: true
I20250905 19:52:14.415302 1170 ts_tablet_manager.cc:1428] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:14.415861 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1a37863c1a764e7eb95e92d19da52a61" } new_table_name: "default.Table_a77e895ae1804560879140ca64db74f6" modify_external_catalogs: true
I20250905 19:52:14.416530 1170 tablet_bootstrap.cc:492] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.417207 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_da585f1b2ba344b2a01227658054e1a1" } modify_external_catalogs: true
I20250905 19:52:14.417376 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_da585f1b2ba344b2a01227658054e1a1" } modify_external_catalogs: true
I20250905 19:52:14.417711 1186 tablet_replica.cc:331] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:14.418069 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5b9c5afac54648e19d47aca2d54aa228" } new_table_name: "default.Table_a5d3b9216c2241189c590a6ec273ac95" modify_external_catalogs: true
I20250905 19:52:14.419085 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_893d44ae3142460cbfbc43ac8756dd02" } modify_external_catalogs: true
I20250905 19:52:14.419312 1171 tablet_bootstrap.cc:492] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.419293 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_893d44ae3142460cbfbc43ac8756dd02" } modify_external_catalogs: true
I20250905 19:52:14.420799 1171 tablet_bootstrap.cc:654] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.420928 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet c7c015508fe7465d852fe3875a8b1aa5 (table default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:14.421216 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c7c015508fe7465d852fe3875a8b1aa5 with delete_type TABLET_DATA_DELETED (replaced by tablet c2a054ea3bca416b8fa344a5de28d798) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.424209 1186 raft_consensus.cc:2241] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.424409 1186 raft_consensus.cc:2270] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.418807 1170 tablet_bootstrap.cc:654] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.420408 1175 raft_consensus.cc:491] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.424762 1175 raft_consensus.cc:513] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.426761 1169 tablet_bootstrap.cc:492] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.426983 1175 leader_election.cc:290] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.432482 1033 tablet_service.cc:1468] Processing CreateTablet for tablet c2a054ea3bca416b8fa344a5de28d798 (DEFAULT_TABLE table=default.Table_fc44097d961e4e8ab5ce95984b17bc16 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.433086 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c2a054ea3bca416b8fa344a5de28d798. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.436555 1186 ts_tablet_manager.cc:1905] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.441668 1169 ts_tablet_manager.cc:1397] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.044s user 0.000s sys 0.004s
I20250905 19:52:14.442952 1169 raft_consensus.cc:357] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.444388 1169 raft_consensus.cc:383] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.444738 1169 raft_consensus.cc:738] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.445225 1169 consensus_queue.cc:260] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.446367 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a5d48075d7bc41f2a35838cbba688260
I20250905 19:52:14.446377 1184 tablet_replica.cc:331] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.446847 1184 raft_consensus.cc:2241] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:14.447134 962 tablet.cc:2378] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:14.447260 1184 raft_consensus.cc:2270] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.447520 1169 ts_tablet_manager.cc:1428] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:14.447854 1169 tablet_bootstrap.cc:492] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.450611 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417199577c6041b5801756876da3448a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:14.451081 1053 raft_consensus.cc:2466] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:14.451511 1184 ts_tablet_manager.cc:1905] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.451838 711 leader_election.cc:304] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:14.452268 1175 raft_consensus.cc:2802] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:14.452415 1175 raft_consensus.cc:491] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:14.452520 1175 raft_consensus.cc:3058] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:14.452943 1101 tablet.cc:2378] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:14.455718 1175 raft_consensus.cc:513] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:14.457881 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e68bdbba23704093a01c0a5b576d3f6d has no consensus state
I20250905 19:52:14.458165 1175 leader_election.cc:290] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.458693 454 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet a4897f796f694522ae960848c4ff8bb7
I20250905 19:52:14.458863 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417199577c6041b5801756876da3448a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:14.459060 1053 raft_consensus.cc:3058] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.459950 1186 ts_tablet_manager.cc:1918] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.460103 1186 log.cc:1199] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4d21be9d7e5140b6aae49e12f0509b12
I20250905 19:52:14.460490 1186 ts_tablet_manager.cc:1939] T 4d21be9d7e5140b6aae49e12f0509b12 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.461616 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417199577c6041b5801756876da3448a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
W20250905 19:52:14.462698 712 leader_election.cc:343] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.462767 1053 raft_consensus.cc:2466] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:14.462997 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "417199577c6041b5801756876da3448a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:14.465265 712 leader_election.cc:343] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.466544 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ca53e56561674c8c830a0e28443a19e0
I20250905 19:52:14.466928 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4d21be9d7e5140b6aae49e12f0509b12 (table default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:14.468266 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ca53e56561674c8c830a0e28443a19e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.468644 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet ca53e56561674c8c830a0e28443a19e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.466538 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 91c7f96dfe02439585e1f16682d89ac4
W20250905 19:52:14.470484 831 tablet.cc:2378] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:14.471743 1171 tablet_bootstrap.cc:492] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.471941 1171 ts_tablet_manager.cc:1397] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.011s
I20250905 19:52:14.472800 1171 raft_consensus.cc:357] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.473021 1171 raft_consensus.cc:383] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.473114 1171 raft_consensus.cc:738] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.473407 1171 consensus_queue.cc:260] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.463466 711 leader_election.cc:304] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:14.475014 756 tablet_service.cc:1515] Processing DeleteTablet for tablet ca53e56561674c8c830a0e28443a19e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.475329 1175 raft_consensus.cc:2802] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:14.476363 1171 ts_tablet_manager.cc:1428] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250905 19:52:14.476760 1171 tablet_bootstrap.cc:492] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.477587 1180 raft_consensus.cc:491] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.477804 1180 raft_consensus.cc:513] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.478483 1169 tablet_bootstrap.cc:654] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.478658 1176 raft_consensus.cc:491] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.478847 1176 raft_consensus.cc:513] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.479786 1180 leader_election.cc:290] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.480144 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_5b9c5afac54648e19d47aca2d54aa228 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 1 got 0
I20250905 19:52:14.480525 1176 leader_election.cc:290] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:14.480867 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1f668b9ae964b7ebf87726f734e3a40 consensus state has no leader
I20250905 19:52:14.482209 893 tablet_service.cc:1468] Processing CreateTablet for tablet 1e2630ff3a4848ada70176448de607e2 (DEFAULT_TABLE table=default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.482656 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e2630ff3a4848ada70176448de607e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:14.483029 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ca53e56561674c8c830a0e28443a19e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca53e56561674c8c830a0e28443a19e0 already in progress: creating tablet
I20250905 19:52:14.483398 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 417199577c6041b5801756876da3448a
I20250905 19:52:14.484113 754 tablet_service.cc:1468] Processing CreateTablet for tablet 17d0a48c25b84facb082670cb898468c (DEFAULT_TABLE table=default.Table_accba7263cc04e558868883f599f2838 [id=59ebabf8b07641e8bc87c8003786c9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.484495 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d0a48c25b84facb082670cb898468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.488519 891 tablet_service.cc:1468] Processing CreateTablet for tablet 17d0a48c25b84facb082670cb898468c (DEFAULT_TABLE table=default.Table_accba7263cc04e558868883f599f2838 [id=59ebabf8b07641e8bc87c8003786c9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.488689 1185 ts_tablet_manager.cc:1918] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.488893 1185 log.cc:1199] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/4d21be9d7e5140b6aae49e12f0509b12
I20250905 19:52:14.489023 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d0a48c25b84facb082670cb898468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.489415 1185 ts_tablet_manager.cc:1939] T 4d21be9d7e5140b6aae49e12f0509b12 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:14.492437 756 tablet_service.cc:1468] Processing CreateTablet for tablet 1e2630ff3a4848ada70176448de607e2 (DEFAULT_TABLE table=default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.492899 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e2630ff3a4848ada70176448de607e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.495179 1171 tablet_bootstrap.cc:654] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.497556 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_d1558a276407476f80051a63eea7bdaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.498185 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d1558a276407476f80051a63eea7bdaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.498308 894 tablet_service.cc:1468] Processing CreateTablet for tablet e68bdbba23704093a01c0a5b576d3f6d (DEFAULT_TABLE table=default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.498449 890 tablet_service.cc:1468] Processing CreateTablet for tablet 987a6193d2a74f90906a45b237040e39 (DEFAULT_TABLE table=default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.498782 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68bdbba23704093a01c0a5b576d3f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.498867 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 417199577c6041b5801756876da3448a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.501044 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987a6193d2a74f90906a45b237040e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.501597 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 417199577c6041b5801756876da3448a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.501569 887 tablet_service.cc:1468] Processing CreateTablet for tablet 266ffb862d144368994f5c33e684647d (DEFAULT_TABLE table=default.Table_06206a701b6e4278a053ea096bff8659 [id=a53613dd2a0a430f83f027b67ee14b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.501855 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 266ffb862d144368994f5c33e684647d (DEFAULT_TABLE table=default.Table_06206a701b6e4278a053ea096bff8659 [id=a53613dd2a0a430f83f027b67ee14b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.502036 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266ffb862d144368994f5c33e684647d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.502070 1030 tablet_service.cc:1468] Processing CreateTablet for tablet e68bdbba23704093a01c0a5b576d3f6d (DEFAULT_TABLE table=default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.502475 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266ffb862d144368994f5c33e684647d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.502624 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 987a6193d2a74f90906a45b237040e39 (DEFAULT_TABLE table=default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.502955 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987a6193d2a74f90906a45b237040e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.504081 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 17d0a48c25b84facb082670cb898468c (DEFAULT_TABLE table=default.Table_accba7263cc04e558868883f599f2838 [id=59ebabf8b07641e8bc87c8003786c9ca]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.504534 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 17d0a48c25b84facb082670cb898468c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.510125 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68bdbba23704093a01c0a5b576d3f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.510957 888 tablet_service.cc:1468] Processing CreateTablet for tablet e25832a05f034c25b20b55adfdcb63d7 (DEFAULT_TABLE table=default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 [id=f483c42f9aea43f38894538f57e7db56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.511586 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e25832a05f034c25b20b55adfdcb63d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.512506 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 4d21be9d7e5140b6aae49e12f0509b12 (table default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:14.513393 1185 tablet_replica.cc:331] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:14.516736 1175 raft_consensus.cc:695] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:14.517315 1175 consensus_queue.cc:237] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.513764 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 417199577c6041b5801756876da3448a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.514302 886 tablet_service.cc:1468] Processing CreateTablet for tablet ffa14750f1b94c589779f7da0e764ccc (DEFAULT_TABLE table=default.Table_e7d692a2d299432090b66b38168b5894 [id=8a1b8eddc57e400696283fed538ae685]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.521613 1196 raft_consensus.cc:491] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.522071 1196 raft_consensus.cc:513] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.523090 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 417199577c6041b5801756876da3448a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.523629 1186 tablet_replica.cc:331] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:14.523767 754 tablet_service.cc:1468] Processing CreateTablet for tablet 987a6193d2a74f90906a45b237040e39 (DEFAULT_TABLE table=default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.524173 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 987a6193d2a74f90906a45b237040e39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.524678 1196 leader_election.cc:290] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.525216 756 tablet_service.cc:1468] Processing CreateTablet for tablet e68bdbba23704093a01c0a5b576d3f6d (DEFAULT_TABLE table=default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.525628 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e68bdbba23704093a01c0a5b576d3f6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.530573 452 master_service.cc:948] ReplaceTablet: received request to replace tablet c2a054ea3bca416b8fa344a5de28d798 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.534341 1170 tablet_bootstrap.cc:492] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.534639 1170 ts_tablet_manager.cc:1397] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.003s
I20250905 19:52:14.534837 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 987a6193d2a74f90906a45b237040e39 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.537252 1170 raft_consensus.cc:357] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.532557 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ffa14750f1b94c589779f7da0e764ccc (DEFAULT_TABLE table=default.Table_e7d692a2d299432090b66b38168b5894 [id=8a1b8eddc57e400696283fed538ae685]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.538101 1028 tablet_service.cc:1468] Processing CreateTablet for tablet e25832a05f034c25b20b55adfdcb63d7 (DEFAULT_TABLE table=default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 [id=f483c42f9aea43f38894538f57e7db56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.540217 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f668b9ae964b7ebf87726f734e3a40" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:14.541071 450 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_b265ba1d975f43fe9d26542d0296d10d [id=95b523055663453a83fe91e35e1205cc]). Expected version 2 got 0
I20250905 19:52:14.541282 1180 raft_consensus.cc:491] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.541458 1180 raft_consensus.cc:513] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.542129 1180 leader_election.cc:290] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.542261 1170 raft_consensus.cc:383] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.542368 1170 raft_consensus.cc:738] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.542871 1184 ts_tablet_manager.cc:1918] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.542680 1170 consensus_queue.cc:260] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.537487 1186 raft_consensus.cc:2241] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:14.540395 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 417199577c6041b5801756876da3448a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.544943 1184 log.cc:1199] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/c7c015508fe7465d852fe3875a8b1aa5
I20250905 19:52:14.546381 1184 ts_tablet_manager.cc:1939] T c7c015508fe7465d852fe3875a8b1aa5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.546622 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa14750f1b94c589779f7da0e764ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.552227 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f668b9ae964b7ebf87726f734e3a40" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:14.553385 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e25832a05f034c25b20b55adfdcb63d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.555238 755 tablet_service.cc:1468] Processing CreateTablet for tablet 266ffb862d144368994f5c33e684647d (DEFAULT_TABLE table=default.Table_06206a701b6e4278a053ea096bff8659 [id=a53613dd2a0a430f83f027b67ee14b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.555275 754 tablet_service.cc:1468] Processing CreateTablet for tablet e25832a05f034c25b20b55adfdcb63d7 (DEFAULT_TABLE table=default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 [id=f483c42f9aea43f38894538f57e7db56]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.555732 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 266ffb862d144368994f5c33e684647d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:14.555954 712 leader_election.cc:343] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.556097 1053 raft_consensus.cc:2466] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:14.556664 711 leader_election.cc:304] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:14.546165 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 1e2630ff3a4848ada70176448de607e2 (DEFAULT_TABLE table=default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.560387 752 tablet_service.cc:1468] Processing CreateTablet for tablet ffa14750f1b94c589779f7da0e764ccc (DEFAULT_TABLE table=default.Table_e7d692a2d299432090b66b38168b5894 [id=8a1b8eddc57e400696283fed538ae685]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.560868 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa14750f1b94c589779f7da0e764ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.560863 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e2630ff3a4848ada70176448de607e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.564532 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet c7c015508fe7465d852fe3875a8b1aa5 (table default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:14.566259 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ffa14750f1b94c589779f7da0e764ccc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.566987 1184 tablet_replica.cc:331] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.568310 1184 raft_consensus.cc:2241] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.568524 1184 raft_consensus.cc:2270] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.571671 446 master_service.cc:948] ReplaceTablet: received request to replace tablet c2a054ea3bca416b8fa344a5de28d798 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.573308 449 master_service.cc:948] ReplaceTablet: received request to replace tablet ca53e56561674c8c830a0e28443a19e0 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.573796 1184 ts_tablet_manager.cc:1905] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.573282 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 987a6193d2a74f90906a45b237040e39 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.577343 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e25832a05f034c25b20b55adfdcb63d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.580776 1184 ts_tablet_manager.cc:1918] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.580953 1184 log.cc:1199] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/4d21be9d7e5140b6aae49e12f0509b12
I20250905 19:52:14.581326 1184 ts_tablet_manager.cc:1939] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.582012 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 4d21be9d7e5140b6aae49e12f0509b12 (table default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:14.582989 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8fe7ea747ff749c389bdd49a8291bf19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.583778 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fe7ea747ff749c389bdd49a8291bf19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.584499 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_637625576ccf4079a93df8d3a547ff9e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.584600 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8cdd361d6e274d5eb920e324573d83fd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.585083 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8cdd361d6e274d5eb920e324573d83fd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.585084 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_637625576ccf4079a93df8d3a547ff9e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.589058 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c865b4a070ba494fab94d0daa544d56f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.589470 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet c2a054ea3bca416b8fa344a5de28d798
W20250905 19:52:14.589571 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c865b4a070ba494fab94d0daa544d56f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.590044 452 catalog_manager.cc:6652] ReplaceTablet: tablet c2a054ea3bca416b8fa344a5de28d798 deleted and replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77
I20250905 19:52:14.590480 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 987a6193d2a74f90906a45b237040e39
I20250905 19:52:14.590966 447 catalog_manager.cc:6652] ReplaceTablet: tablet 987a6193d2a74f90906a45b237040e39 deleted and replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef
I20250905 19:52:14.591614 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_3863927d1ffa41289c4ccd14dcec05f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.592166 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3863927d1ffa41289c4ccd14dcec05f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.596513 984 leader_election.cc:343] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 4d21be9d7e5140b6aae49e12f0509b12
W20250905 19:52:14.596581 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ca53e56561674c8c830a0e28443a19e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca53e56561674c8c830a0e28443a19e0 already in progress: creating tablet
I20250905 19:52:14.596509 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9a82fd7e6c07481ba6c70efeab05238b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.597023 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 987a6193d2a74f90906a45b237040e39 with delete_type TABLET_DATA_DELETED (replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:14.597267 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a82fd7e6c07481ba6c70efeab05238b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.597273 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.597295 1184 tablet_replica.cc:331] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.597669 1184 raft_consensus.cc:2241] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.597877 1184 raft_consensus.cc:2270] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.598726 1196 raft_consensus.cc:2802] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:14.599381 1170 ts_tablet_manager.cc:1428] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.063s user 0.001s sys 0.001s
I20250905 19:52:14.599714 1170 tablet_bootstrap.cc:492] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.599407 1196 raft_consensus.cc:491] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:14.602226 1196 raft_consensus.cc:3058] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:14.603642 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:41317, user_credentials={real_user=slave}} blocked reactor thread for 51836.9us
I20250905 19:52:14.604563 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 987a6193d2a74f90906a45b237040e39 with delete_type TABLET_DATA_DELETED (replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.604934 755 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.605108 1196 raft_consensus.cc:513] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.605840 1186 pending_rounds.cc:70] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
I20250905 19:52:14.606066 1186 pending_rounds.cc:77] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089523788382208 op_type: NO_OP noop_request { }
I20250905 19:52:14.606243 1186 raft_consensus.cc:2887] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:14.606354 1186 raft_consensus.cc:2270] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.607297 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f668b9ae964b7ebf87726f734e3a40" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:14.607498 1053 raft_consensus.cc:3058] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.608489 1169 tablet_bootstrap.cc:492] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.608660 1169 ts_tablet_manager.cc:1397] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.161s user 0.004s sys 0.004s
I20250905 19:52:14.609516 1169 raft_consensus.cc:357] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.609707 1169 raft_consensus.cc:383] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.609817 1169 raft_consensus.cc:738] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.610129 1169 consensus_queue.cc:260] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.610852 1053 raft_consensus.cc:2466] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:14.611129 1196 leader_election.cc:290] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.612049 711 leader_election.cc:304] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:14.612493 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e688c1faf5c84ad1aa446711ae39d273"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.612697 1175 raft_consensus.cc:2802] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:14.612839 1175 raft_consensus.cc:695] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
W20250905 19:52:14.613116 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e688c1faf5c84ad1aa446711ae39d273 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.613129 1175 consensus_queue.cc:237] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.614403 1169 ts_tablet_manager.cc:1428] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 19:52:14.614706 1169 tablet_bootstrap.cc:492] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.616354 1186 ts_tablet_manager.cc:1905] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.616528 1169 tablet_bootstrap.cc:654] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.616508 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_06206a701b6e4278a053ea096bff8659" } new_table_name: "default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e" modify_external_catalogs: true
I20250905 19:52:14.619007 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_34947bac54164aa9bb16b6e3e4760027"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.619704 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_34947bac54164aa9bb16b6e3e4760027 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.620514 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f681ca53501741449436b37d7ee502c7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.621165 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f681ca53501741449436b37d7ee502c7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.621253 1170 tablet_bootstrap.cc:654] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.624826 1186 ts_tablet_manager.cc:1918] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:14.625008 1186 log.cc:1199] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/417199577c6041b5801756876da3448a
I20250905 19:52:14.626022 1184 ts_tablet_manager.cc:1905] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.631141 1186 ts_tablet_manager.cc:1939] T 417199577c6041b5801756876da3448a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.633772 1170 tablet_bootstrap.cc:492] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.633962 1170 ts_tablet_manager.cc:1397] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.003s
I20250905 19:52:14.634944 1170 raft_consensus.cc:357] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.635187 1170 raft_consensus.cc:383] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.635298 1170 raft_consensus.cc:738] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.635562 1170 consensus_queue.cc:260] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.637745 1170 ts_tablet_manager.cc:1428] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250905 19:52:14.638082 1170 tablet_bootstrap.cc:492] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:14.638113 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 417199577c6041b5801756876da3448a because the tablet was not found. No further retry: Not found: Tablet not found: 417199577c6041b5801756876da3448a
W20250905 19:52:14.638568 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 987a6193d2a74f90906a45b237040e39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 987a6193d2a74f90906a45b237040e39 already in progress: creating tablet
I20250905 19:52:14.640264 1186 tablet_replica.cc:331] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:14.640511 1186 raft_consensus.cc:2241] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.640717 1186 raft_consensus.cc:2270] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.641786 1170 tablet_bootstrap.cc:654] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.644605 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a5d3b9216c2241189c590a6ec273ac95" } new_table_name: "default.Table_7ea4506d60df44b8b9a7f137911b5570" modify_external_catalogs: true
I20250905 19:52:14.644781 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_accba7263cc04e558868883f599f2838" } new_table_name: "default.Table_fe362aa06bd64814853c72f1a41c76b3" modify_external_catalogs: true
I20250905 19:52:14.647985 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet ca53e56561674c8c830a0e28443a19e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.649351 1028 tablet_service.cc:1468] Processing CreateTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef (DEFAULT_TABLE table=default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.649811 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f842c6ac96ec44ffaae02edb4a9fdeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.652420 1032 tablet_service.cc:1468] Processing CreateTablet for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (DEFAULT_TABLE table=default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.652599 1186 ts_tablet_manager.cc:1905] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.652777 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec810e5cc1af4b9d9f6bd421072bbc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.657069 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b265ba1d975f43fe9d26542d0296d10d" } modify_external_catalogs: true
I20250905 19:52:14.657349 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b265ba1d975f43fe9d26542d0296d10d" } modify_external_catalogs: true
I20250905 19:52:14.661360 1184 ts_tablet_manager.cc:1918] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.661518 1184 log.cc:1199] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/417199577c6041b5801756876da3448a
I20250905 19:52:14.661536 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3863927d1ffa41289c4ccd14dcec05f9" } modify_external_catalogs: true
I20250905 19:52:14.661787 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3863927d1ffa41289c4ccd14dcec05f9" } modify_external_catalogs: true
I20250905 19:52:14.661929 1184 ts_tablet_manager.cc:1939] T 417199577c6041b5801756876da3448a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.662659 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9a82fd7e6c07481ba6c70efeab05238b" } new_table_name: "default.Table_056e9cf230e44fc5badd0c9b709c0c4f" modify_external_catalogs: true
I20250905 19:52:14.662940 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 417199577c6041b5801756876da3448a (table default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]) successfully deleted
I20250905 19:52:14.667353 1185 raft_consensus.cc:2241] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.668048 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ca53e56561674c8c830a0e28443a19e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.668283 755 tablet_service.cc:1468] Processing CreateTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef (DEFAULT_TABLE table=default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.668545 756 tablet_service.cc:1468] Processing CreateTablet for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (DEFAULT_TABLE table=default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.668740 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f842c6ac96ec44ffaae02edb4a9fdeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.669006 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec810e5cc1af4b9d9f6bd421072bbc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.677920 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d1558a276407476f80051a63eea7bdaf" } modify_external_catalogs: true
I20250905 19:52:14.678151 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d1558a276407476f80051a63eea7bdaf" } modify_external_catalogs: true
I20250905 19:52:14.680421 1186 ts_tablet_manager.cc:1918] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.680606 1186 log.cc:1199] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/c2a054ea3bca416b8fa344a5de28d798
I20250905 19:52:14.681115 1186 ts_tablet_manager.cc:1939] T c2a054ea3bca416b8fa344a5de28d798 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.681769 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 987a6193d2a74f90906a45b237040e39 with delete_type TABLET_DATA_DELETED (replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:14.682052 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet c2a054ea3bca416b8fa344a5de28d798 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2a054ea3bca416b8fa344a5de28d798 already in progress: creating tablet
W20250905 19:52:14.682390 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 987a6193d2a74f90906a45b237040e39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 987a6193d2a74f90906a45b237040e39 already in progress: creating tablet
I20250905 19:52:14.682893 894 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.682960 1170 tablet_bootstrap.cc:492] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.682998 1169 tablet_bootstrap.cc:492] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.683133 1169 ts_tablet_manager.cc:1397] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.069s user 0.005s sys 0.020s
I20250905 19:52:14.683127 1170 ts_tablet_manager.cc:1397] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.001s
I20250905 19:52:14.683270 1185 raft_consensus.cc:2270] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.683929 1169 raft_consensus.cc:357] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.684124 1184 tablet_replica.cc:331] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.684203 1169 raft_consensus.cc:383] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.684288 1169 raft_consensus.cc:738] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.684330 1184 raft_consensus.cc:2241] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.684516 1184 raft_consensus.cc:2270] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.684975 1186 tablet_replica.cc:331] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:14.684983 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d1f668b9ae964b7ebf87726f734e3a40 consensus state has no leader
I20250905 19:52:14.685169 1186 raft_consensus.cc:2241] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.685320 1186 raft_consensus.cc:2270] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.686022 1186 ts_tablet_manager.cc:1905] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.686094 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3db28a103a6f46f08d3aeaf8fb3432d5" } modify_external_catalogs: true
I20250905 19:52:14.686288 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3db28a103a6f46f08d3aeaf8fb3432d5" } modify_external_catalogs: true
W20250905 19:52:14.686868 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 266ffb862d144368994f5c33e684647d consensus state has no leader
I20250905 19:52:14.684525 1169 consensus_queue.cc:260] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.687709 1169 ts_tablet_manager.cc:1428] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:14.687949 1185 ts_tablet_manager.cc:1905] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.688326 1169 tablet_bootstrap.cc:492] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:14.688766 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 17d0a48c25b84facb082670cb898468c consensus state has no leader
I20250905 19:52:14.690362 1169 tablet_bootstrap.cc:654] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.691330 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet c2a054ea3bca416b8fa344a5de28d798 (table default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:14.691794 1184 ts_tablet_manager.cc:1905] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.692821 894 tablet_service.cc:1468] Processing CreateTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef (DEFAULT_TABLE table=default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.693012 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_637625576ccf4079a93df8d3a547ff9e" } new_table_name: "default.Table_c32f9552869b4ea686b824e1967c932c" modify_external_catalogs: true
I20250905 19:52:14.693271 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f842c6ac96ec44ffaae02edb4a9fdeef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.696261 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1f668b9ae964b7ebf87726f734e3a40" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:14.696535 1170 raft_consensus.cc:357] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.696755 1170 raft_consensus.cc:383] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.696846 1170 raft_consensus.cc:738] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:14.697014 712 leader_election.cc:343] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.697387 888 tablet_service.cc:1468] Processing CreateTablet for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (DEFAULT_TABLE table=default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.697328 1170 consensus_queue.cc:260] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.698093 1170 ts_tablet_manager.cc:1428] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.015s user 0.002s sys 0.001s
I20250905 19:52:14.698259 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_34947bac54164aa9bb16b6e3e4760027" } modify_external_catalogs: true
I20250905 19:52:14.698385 1170 tablet_bootstrap.cc:492] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.698443 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_34947bac54164aa9bb16b6e3e4760027" } modify_external_catalogs: true
I20250905 19:52:14.700013 1170 tablet_bootstrap.cc:654] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.703588 1186 ts_tablet_manager.cc:1918] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.703729 1186 log.cc:1199] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ca53e56561674c8c830a0e28443a19e0
I20250905 19:52:14.704442 1169 tablet_bootstrap.cc:492] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.704619 1169 ts_tablet_manager.cc:1397] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20250905 19:52:14.705335 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ec810e5cc1af4b9d9f6bd421072bbc77. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.705420 1169 raft_consensus.cc:357] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.705610 1169 raft_consensus.cc:383] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.705682 1169 raft_consensus.cc:738] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.705976 1169 consensus_queue.cc:260] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:14.707957 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet eacedd2e38d64d2cb25b2ebd88f563a1 has no consensus state
I20250905 19:52:14.708592 1169 ts_tablet_manager.cc:1428] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:14.709671 455 catalog_manager.cc:5582] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:14.710760 455 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_7ea4506d60df44b8b9a7f137911b5570 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 3 got 0
I20250905 19:52:14.714897 1169 tablet_bootstrap.cc:492] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.716540 891 tablet_service.cc:1468] Processing CreateTablet for tablet e5a7c8a1fe604706be88d0461b241a31 (DEFAULT_TABLE table=default.Table_c865b4a070ba494fab94d0daa544d56f [id=53cc2a672c8b437ea0540508da565731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.716696 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 7e62dbe9e48544109446323b2cc4501f (DEFAULT_TABLE table=default.Table_3863927d1ffa41289c4ccd14dcec05f9 [id=567a4196a6af4e2b828347357c44313c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.716928 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a7c8a1fe604706be88d0461b241a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.716976 1169 tablet_bootstrap.cc:654] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.717136 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e62dbe9e48544109446323b2cc4501f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.718195 892 tablet_service.cc:1468] Processing CreateTablet for tablet afb404e5b06c452bb98abeea95070a5f (DEFAULT_TABLE table=default.Table_637625576ccf4079a93df8d3a547ff9e [id=b1e354dd48554016baae491637511fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.719050 1106 raft_consensus.cc:491] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c401c9f3020b4ae5931c566c6bcb1711)
I20250905 19:52:14.719244 1106 raft_consensus.cc:513] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:14.719915 1106 leader_election.cc:290] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317)
I20250905 19:52:14.720172 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb404e5b06c452bb98abeea95070a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.720259 1032 tablet_service.cc:1468] Processing CreateTablet for tablet e5a7c8a1fe604706be88d0461b241a31 (DEFAULT_TABLE table=default.Table_c865b4a070ba494fab94d0daa544d56f [id=53cc2a672c8b437ea0540508da565731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.721190 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a7c8a1fe604706be88d0461b241a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.721256 1184 ts_tablet_manager.cc:1918] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.721395 1184 log.cc:1199] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ca53e56561674c8c830a0e28443a19e0
I20250905 19:52:14.721280 893 tablet_service.cc:1468] Processing CreateTablet for tablet cbf0b20630b44230b17a80cdb53322fa (DEFAULT_TABLE table=default.Table_8cdd361d6e274d5eb920e324573d83fd [id=3ce5d572e4104f8cb8100e115eaa8df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.721791 1184 ts_tablet_manager.cc:1939] T ca53e56561674c8c830a0e28443a19e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.721900 887 tablet_service.cc:1468] Processing CreateTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f (DEFAULT_TABLE table=default.Table_d1558a276407476f80051a63eea7bdaf [id=ff84de6d110a451c9b8da5ce3803b1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.722934 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ca53e56561674c8c830a0e28443a19e0 (table default.Table_0a4b61de931546689e038111b0a405a4 [id=b72b466b7abe468dbaa985c9f469cbcf]) successfully deleted
I20250905 19:52:14.723644 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu with pid 564
I20250905 19:52:14.724269 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f (DEFAULT_TABLE table=default.Table_d1558a276407476f80051a63eea7bdaf [id=ff84de6d110a451c9b8da5ce3803b1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.724629 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8240e51124a74b81a0bdabb3ef35e66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.725755 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:14.726382 890 tablet_service.cc:1468] Processing CreateTablet for tablet 7e62dbe9e48544109446323b2cc4501f (DEFAULT_TABLE table=default.Table_3863927d1ffa41289c4ccd14dcec05f9 [id=567a4196a6af4e2b828347357c44313c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.726773 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf0b20630b44230b17a80cdb53322fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.727866 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8240e51124a74b81a0bdabb3ef35e66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.728125 1169 tablet_bootstrap.cc:492] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.728274 1169 ts_tablet_manager.cc:1397] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.005s
I20250905 19:52:14.728312 1027 tablet_service.cc:1468] Processing CreateTablet for tablet cbf0b20630b44230b17a80cdb53322fa (DEFAULT_TABLE table=default.Table_8cdd361d6e274d5eb920e324573d83fd [id=3ce5d572e4104f8cb8100e115eaa8df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.728684 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf0b20630b44230b17a80cdb53322fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.728701 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e62dbe9e48544109446323b2cc4501f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.729203 1169 raft_consensus.cc:357] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.729434 1169 raft_consensus.cc:383] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.729561 1169 raft_consensus.cc:738] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.729847 1169 consensus_queue.cc:260] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.730412 1169 ts_tablet_manager.cc:1428] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250905 19:52:14.732030 849 leader_election.cc:343] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 417199577c6041b5801756876da3448a
I20250905 19:52:14.731998 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92ea9fe1c36c4807ba02f23e6877198c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:14.732311 1052 raft_consensus.cc:2466] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:14.732501 1169 tablet_bootstrap.cc:492] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.733966 1169 tablet_bootstrap.cc:654] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.737161 849 leader_election.cc:304] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:14.737632 1180 raft_consensus.cc:2802] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:14.737762 1180 raft_consensus.cc:491] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:14.737884 1180 raft_consensus.cc:3058] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:14.740325 986 leader_election.cc:343] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 4d21be9d7e5140b6aae49e12f0509b12
I20250905 19:52:14.740535 986 leader_election.cc:304] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:14.740819 1180 raft_consensus.cc:513] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:14.740922 986 raft_consensus.cc:2684] T 4d21be9d7e5140b6aae49e12f0509b12 P 4eb3af79b01d4835bf6bf7b26bf016f5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:14.741489 1180 leader_election.cc:290] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.742208 1169 tablet_bootstrap.cc:492] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.742378 1169 ts_tablet_manager.cc:1397] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20250905 19:52:14.742648 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92ea9fe1c36c4807ba02f23e6877198c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:14.742843 1052 raft_consensus.cc:3058] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.746304 1052 raft_consensus.cc:2466] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:14.746889 849 leader_election.cc:304] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:14.747345 1180 raft_consensus.cc:2802] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:14.749373 1185 ts_tablet_manager.cc:1918] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.749527 1185 log.cc:1199] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/c7c015508fe7465d852fe3875a8b1aa5
I20250905 19:52:14.749958 1185 ts_tablet_manager.cc:1939] T c7c015508fe7465d852fe3875a8b1aa5 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:14.749881 1169 raft_consensus.cc:357] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.750234 1169 raft_consensus.cc:383] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.750363 1169 raft_consensus.cc:738] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.750391 1030 tablet_service.cc:1468] Processing CreateTablet for tablet afb404e5b06c452bb98abeea95070a5f (DEFAULT_TABLE table=default.Table_637625576ccf4079a93df8d3a547ff9e [id=b1e354dd48554016baae491637511fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.750873 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb404e5b06c452bb98abeea95070a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.751039 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet c7c015508fe7465d852fe3875a8b1aa5 (table default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
W20250905 19:52:14.751784 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ca53e56561674c8c830a0e28443a19e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ca53e56561674c8c830a0e28443a19e0 already in progress: creating tablet
I20250905 19:52:14.752126 1185 tablet_replica.cc:331] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:14.752317 1185 raft_consensus.cc:2241] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.752497 1185 raft_consensus.cc:2270] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:14.753762 848 leader_election.cc:343] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 417199577c6041b5801756876da3448a
I20250905 19:52:14.753904 848 leader_election.cc:304] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:14.754132 848 raft_consensus.cc:2684] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:14.754724 535 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 3 candidate_status { last_received { term: 2 index: 15 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" is_pre_election: true
I20250905 19:52:14.756556 886 tablet_service.cc:1468] Processing CreateTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 (DEFAULT_TABLE table=default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.756973 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9b3f29d521431f9eb8ea06ece31552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.757293 1171 tablet_bootstrap.cc:492] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.757428 1171 ts_tablet_manager.cc:1397] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.281s user 0.048s sys 0.061s
I20250905 19:52:14.758208 1185 ts_tablet_manager.cc:1905] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.758523 1169 consensus_queue.cc:260] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.759431 1169 ts_tablet_manager.cc:1428] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250905 19:52:14.760309 1169 tablet_bootstrap.cc:492] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:14.762413 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:41317, user_credentials={real_user=slave}} blocked reactor thread for 57107.3us
I20250905 19:52:14.758989 1171 raft_consensus.cc:357] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.765666 1171 raft_consensus.cc:383] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.765777 1171 raft_consensus.cc:738] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.704082 1186 ts_tablet_manager.cc:1939] T ca53e56561674c8c830a0e28443a19e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.717875 1175 raft_consensus.cc:491] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.769371 1175 raft_consensus.cc:513] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.769870 1171 consensus_queue.cc:260] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.770164 1175 leader_election.cc:290] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:14.724583 1170 tablet_bootstrap.cc:492] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.770435 1170 ts_tablet_manager.cc:1397] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.072s user 0.003s sys 0.000s
I20250905 19:52:14.771262 1170 raft_consensus.cc:357] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.771447 1170 raft_consensus.cc:383] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.771543 1170 raft_consensus.cc:738] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.771739 1170 consensus_queue.cc:260] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.771881 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f (DEFAULT_TABLE table=default.Table_d1558a276407476f80051a63eea7bdaf [id=ff84de6d110a451c9b8da5ce3803b1f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.772287 1170 ts_tablet_manager.cc:1428] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:14.774212 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92ea9fe1c36c4807ba02f23e6877198c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:14.774421 751 tablet_service.cc:1468] Processing CreateTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 (DEFAULT_TABLE table=default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.768690 754 tablet_service.cc:1468] Processing CreateTablet for tablet e5a7c8a1fe604706be88d0461b241a31 (DEFAULT_TABLE table=default.Table_c865b4a070ba494fab94d0daa544d56f [id=53cc2a672c8b437ea0540508da565731]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.763355 756 tablet_service.cc:1468] Processing CreateTablet for tablet 7e62dbe9e48544109446323b2cc4501f (DEFAULT_TABLE table=default.Table_3863927d1ffa41289c4ccd14dcec05f9 [id=567a4196a6af4e2b828347357c44313c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.772611 1170 tablet_bootstrap.cc:492] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.772336 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8240e51124a74b81a0bdabb3ef35e66f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.773011 752 tablet_service.cc:1468] Processing CreateTablet for tablet cbf0b20630b44230b17a80cdb53322fa (DEFAULT_TABLE table=default.Table_8cdd361d6e274d5eb920e324573d83fd [id=3ce5d572e4104f8cb8100e115eaa8df0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.775347 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5a7c8a1fe604706be88d0461b241a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.775534 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbf0b20630b44230b17a80cdb53322fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.753682 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92ea9fe1c36c4807ba02f23e6877198c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:14.776419 775 raft_consensus.cc:3058] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.777467 1170 tablet_bootstrap.cc:654] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.779551 775 raft_consensus.cc:2466] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:14.779619 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afb404e5b06c452bb98abeea95070a5f consensus state has no leader
I20250905 19:52:14.779906 451 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 7c87f05721db4e11b8b2036ca5a7286e
I20250905 19:52:14.781935 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 8240e51124a74b81a0bdabb3ef35e66f
I20250905 19:52:14.782567 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 92ea9fe1c36c4807ba02f23e6877198c
I20250905 19:52:14.783171 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet e25832a05f034c25b20b55adfdcb63d7
I20250905 19:52:14.753139 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92ea9fe1c36c4807ba02f23e6877198c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:14.783341 776 raft_consensus.cc:2374] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 835daf30e4324782848290be181ae78f in term 1. Re-sending same reply.
I20250905 19:52:14.773985 753 tablet_service.cc:1468] Processing CreateTablet for tablet afb404e5b06c452bb98abeea95070a5f (DEFAULT_TABLE table=default.Table_637625576ccf4079a93df8d3a547ff9e [id=b1e354dd48554016baae491637511fa7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.783742 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 7e62dbe9e48544109446323b2cc4501f
I20250905 19:52:14.783946 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet afb404e5b06c452bb98abeea95070a5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.785851 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e25832a05f034c25b20b55adfdcb63d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.786140 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.786368 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.788152 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e62dbe9e48544109446323b2cc4501f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.788290 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e62dbe9e48544109446323b2cc4501f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.788360 447 master_service.cc:948] ReplaceTablet: received request to replace tablet e68bdbba23704093a01c0a5b576d3f6d from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.794238 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9b3f29d521431f9eb8ea06ece31552. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:14.795198 513 connection.cc:537] server connection from 127.0.0.1:52534 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:14.795707 439 connection.cc:537] server connection from 127.0.0.1:52576 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:14.799271 441 consensus_peers.cc:489] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 -> Peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Couldn't send request to peer e38333af29db4bf6af41ee5a25d6701e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:14.800112 1170 tablet_bootstrap.cc:492] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.800273 1170 ts_tablet_manager.cc:1397] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.002s
W20250905 19:52:14.800923 959 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:34395 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:34395 (error 108)
I20250905 19:52:14.801014 1170 raft_consensus.cc:357] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.801208 1170 raft_consensus.cc:383] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.801308 1170 raft_consensus.cc:738] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:14.802356 821 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:34395 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:34395 (error 108)
I20250905 19:52:14.802237 1170 consensus_queue.cc:260] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.802738 1169 tablet_bootstrap.cc:654] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.803022 1176 raft_consensus.cc:491] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.803025 1185 ts_tablet_manager.cc:1918] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.803159 1185 log.cc:1199] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/417199577c6041b5801756876da3448a
I20250905 19:52:14.803527 1185 ts_tablet_manager.cc:1939] T 417199577c6041b5801756876da3448a P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:14.803907 1170 ts_tablet_manager.cc:1428] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:14.804226 1170 tablet_bootstrap.cc:492] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.804616 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 417199577c6041b5801756876da3448a (table default.Table_02fc89d2523848a4aba495c72a35491f [id=6c3d3cc15d0e4958bb291fcdd5e292cf]) successfully deleted
I20250905 19:52:14.805244 1185 tablet_replica.cc:331] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:14.805315 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 987a6193d2a74f90906a45b237040e39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 987a6193d2a74f90906a45b237040e39 already in progress: creating tablet
W20250905 19:52:14.805531 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet c2a054ea3bca416b8fa344a5de28d798 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c2a054ea3bca416b8fa344a5de28d798 already in progress: creating tablet
W20250905 19:52:14.805776 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet e25832a05f034c25b20b55adfdcb63d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e25832a05f034c25b20b55adfdcb63d7 already in progress: creating tablet
W20250905 19:52:14.806326 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8240e51124a74b81a0bdabb3ef35e66f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8240e51124a74b81a0bdabb3ef35e66f already in progress: creating tablet
I20250905 19:52:14.807811 448 master_service.cc:948] ReplaceTablet: received request to replace tablet ef9b3f29d521431f9eb8ea06ece31552 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.808818 1176 raft_consensus.cc:513] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.809474 1176 leader_election.cc:290] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:14.812899 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.813402 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 (DEFAULT_TABLE table=default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:14.812918 984 leader_election.cc:343] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: c2a054ea3bca416b8fa344a5de28d798
W20250905 19:52:14.813853 1098 heartbeater.cc:646] Failed to heartbeat to 127.0.104.60:34395 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.104.60:34395 (error 108)
I20250905 19:52:14.814178 1170 tablet_bootstrap.cc:654] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.814209 1171 ts_tablet_manager.cc:1428] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.057s user 0.001s sys 0.009s
I20250905 19:52:14.814699 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7dd83ce427b9438990eee889da0bcb0f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.815315 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7dd83ce427b9438990eee889da0bcb0f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.815516 1171 tablet_bootstrap.cc:492] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.816507 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 1e2630ff3a4848ada70176448de607e2 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.816896 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.817739 1184 tablet_replica.cc:331] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.817945 1184 raft_consensus.cc:2241] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.818555 1171 tablet_bootstrap.cc:654] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.818933 1184 raft_consensus.cc:2270] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.819362 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ef9b3f29d521431f9eb8ea06ece31552. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.819746 1184 ts_tablet_manager.cc:1905] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.822060 455 master_service.cc:948] ReplaceTablet: received request to replace tablet e5a7c8a1fe604706be88d0461b241a31 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:14.822563 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "92ea9fe1c36c4807ba02f23e6877198c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:14.822887 1052 raft_consensus.cc:2391] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:14.825150 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c2a054ea3bca416b8fa344a5de28d798" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:14.827294 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 1e2630ff3a4848ada70176448de607e2 from {username='slave'} at 127.0.0.1:52586
W20250905 19:52:14.827509 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 8240e51124a74b81a0bdabb3ef35e66f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8240e51124a74b81a0bdabb3ef35e66f already in progress: creating tablet
W20250905 19:52:14.827960 986 leader_election.cc:343] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.828323 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet e25832a05f034c25b20b55adfdcb63d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.828591 986 leader_election.cc:304] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:14.828804 986 raft_consensus.cc:2684] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:14.829015 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e62dbe9e48544109446323b2cc4501f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.829268 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.832504 1170 tablet_bootstrap.cc:492] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.832683 1170 ts_tablet_manager.cc:1397] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.029s user 0.012s sys 0.008s
I20250905 19:52:14.833541 1171 tablet_bootstrap.cc:492] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.833542 1170 raft_consensus.cc:357] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.833731 1170 raft_consensus.cc:383] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.833837 1170 raft_consensus.cc:738] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.834028 1171 ts_tablet_manager.cc:1397] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.004s
I20250905 19:52:14.834059 1170 consensus_queue.cc:260] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.834226 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.834775 1171 raft_consensus.cc:357] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.834882 1180 raft_consensus.cc:695] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:14.835585 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.834982 1171 raft_consensus.cc:383] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.836244 1171 raft_consensus.cc:738] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.836561 1171 consensus_queue.cc:260] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.836656 1184 ts_tablet_manager.cc:1918] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.836783 1184 log.cc:1199] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/c2a054ea3bca416b8fa344a5de28d798
I20250905 19:52:14.837642 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ca53e56561674c8c830a0e28443a19e0 (table default.Table_0a4b61de931546689e038111b0a405a4 [id=b72b466b7abe468dbaa985c9f469cbcf]) successfully deleted
I20250905 19:52:14.837659 756 tablet_service.cc:1515] Processing DeleteTablet for tablet e25832a05f034c25b20b55adfdcb63d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.837836 1184 ts_tablet_manager.cc:1939] T c2a054ea3bca416b8fa344a5de28d798 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.837877 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.838014 1171 ts_tablet_manager.cc:1428] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:14.838150 1186 tablet_replica.cc:331] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:14.838327 1171 tablet_bootstrap.cc:492] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.838318 1186 raft_consensus.cc:2241] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.838482 1186 raft_consensus.cc:2270] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.839069 1186 ts_tablet_manager.cc:1905] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.839687 1184 tablet_replica.cc:331] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.839951 1171 tablet_bootstrap.cc:654] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.840284 1184 raft_consensus.cc:2241] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.840444 1184 raft_consensus.cc:2270] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.841203 1184 ts_tablet_manager.cc:1905] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.845921 1186 ts_tablet_manager.cc:1918] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.846131 1186 log.cc:1199] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/92ea9fe1c36c4807ba02f23e6877198c
I20250905 19:52:14.846567 1186 ts_tablet_manager.cc:1939] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.846994 1170 ts_tablet_manager.cc:1428] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.014s user 0.000s sys 0.002s
I20250905 19:52:14.847390 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e62dbe9e48544109446323b2cc4501f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.847851 1184 ts_tablet_manager.cc:1918] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.848050 1184 log.cc:1199] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/92ea9fe1c36c4807ba02f23e6877198c
I20250905 19:52:14.848402 1184 ts_tablet_manager.cc:1939] T 92ea9fe1c36c4807ba02f23e6877198c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.848647 1169 tablet_bootstrap.cc:492] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.848834 1169 ts_tablet_manager.cc:1397] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.002s
I20250905 19:52:14.849622 1169 raft_consensus.cc:357] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.849794 1169 raft_consensus.cc:383] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.849890 1169 raft_consensus.cc:738] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.850121 1169 consensus_queue.cc:260] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.851227 1170 tablet_bootstrap.cc:492] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.852236 1169 ts_tablet_manager.cc:1428] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:14.854192 1169 tablet_bootstrap.cc:492] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.854624 1170 tablet_bootstrap.cc:654] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.855577 1169 tablet_bootstrap.cc:654] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.836692 1180 consensus_queue.cc:237] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.838900 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet c2a054ea3bca416b8fa344a5de28d798 (table default.Table_01b46213458147cd9bf884185bcff837 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:14.858079 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ce8e455023774d02ba896de13623c0ea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.858469 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet e25832a05f034c25b20b55adfdcb63d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e25832a05f034c25b20b55adfdcb63d7 already in progress: creating tablet
W20250905 19:52:14.858728 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 7e62dbe9e48544109446323b2cc4501f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e62dbe9e48544109446323b2cc4501f already in progress: creating tablet
I20250905 19:52:14.858990 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_b265ba1d975f43fe9d26542d0296d10d [id=95b523055663453a83fe91e35e1205cc]) successfully deleted
I20250905 19:52:14.859305 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_dc1ad5f85ed74d81a6bd534a572dc98b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.859768 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dc1ad5f85ed74d81a6bd534a572dc98b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.860327 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1eaa6c2091ee4d949f145d0e9e806550"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.860805 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1eaa6c2091ee4d949f145d0e9e806550 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.858749 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce8e455023774d02ba896de13623c0ea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:14.863442 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet c2a054ea3bca416b8fa344a5de28d798 because the tablet was not found. No further retry: Not found: Tablet not found: c2a054ea3bca416b8fa344a5de28d798
I20250905 19:52:14.864910 1170 tablet_bootstrap.cc:492] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:14.865129 1170 ts_tablet_manager.cc:1397] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.001s
W20250905 19:52:14.865206 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8240e51124a74b81a0bdabb3ef35e66f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8240e51124a74b81a0bdabb3ef35e66f already in progress: creating tablet
W20250905 19:52:14.865420 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e25832a05f034c25b20b55adfdcb63d7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e25832a05f034c25b20b55adfdcb63d7 already in progress: creating tablet
I20250905 19:52:14.865576 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_b265ba1d975f43fe9d26542d0296d10d [id=95b523055663453a83fe91e35e1205cc]) successfully deleted
I20250905 19:52:14.866245 1170 raft_consensus.cc:357] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.866554 1170 raft_consensus.cc:383] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.866678 1169 tablet_bootstrap.cc:492] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:14.866657 1170 raft_consensus.cc:738] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.866918 1169 ts_tablet_manager.cc:1397] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.000s
W20250905 19:52:14.867488 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 7e62dbe9e48544109446323b2cc4501f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e62dbe9e48544109446323b2cc4501f already in progress: creating tablet
I20250905 19:52:14.868577 1169 raft_consensus.cc:357] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.868892 1169 raft_consensus.cc:383] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.868999 1169 raft_consensus.cc:738] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.869315 1169 consensus_queue.cc:260] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.870357 1052 raft_consensus.cc:1273] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:14.871169 1175 consensus_queue.cc:1035] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:14.874029 1169 ts_tablet_manager.cc:1428] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 19:52:14.875737 1185 raft_consensus.cc:2241] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:14.875854 712 leader_election.cc:304] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:14.876144 712 raft_consensus.cc:2684] T 92ea9fe1c36c4807ba02f23e6877198c P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:14.876562 1185 pending_rounds.cc:70] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:14.876755 1185 pending_rounds.cc:77] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089525203206144 op_type: NO_OP noop_request { }
I20250905 19:52:14.876945 1185 raft_consensus.cc:2887] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:14.877082 1185 raft_consensus.cc:2270] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.878319 1169 tablet_bootstrap.cc:492] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:14.879572 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet e68bdbba23704093a01c0a5b576d3f6d
I20250905 19:52:14.880192 447 catalog_manager.cc:6652] ReplaceTablet: tablet e68bdbba23704093a01c0a5b576d3f6d deleted and replaced by tablet d23d6df63c1a4e02a43bccf625e200bb
I20250905 19:52:14.880378 886 tablet_service.cc:1515] Processing DeleteTablet for tablet e68bdbba23704093a01c0a5b576d3f6d with delete_type TABLET_DATA_DELETED (replaced by tablet d23d6df63c1a4e02a43bccf625e200bb) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.880548 1169 tablet_bootstrap.cc:654] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.880733 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ef9b3f29d521431f9eb8ea06ece31552
I20250905 19:52:14.881203 448 catalog_manager.cc:6652] ReplaceTablet: tablet ef9b3f29d521431f9eb8ea06ece31552 deleted and replaced by tablet 8adc74a357244102801b2aa8d27d9747
W20250905 19:52:14.882159 1227 consensus_queue.cc:1162] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f [LEADER]: Queue is closed or peer was untracked, disregarding peer response. Response: responder_uuid: "835daf30e4324782848290be181ae78f" status { last_received { term: 1 index: 1 } last_committed_idx: 0 last_received_current_leader { term: 1 index: 1 } }
I20250905 19:52:14.884660 886 tablet_service.cc:1515] Processing DeleteTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 with delete_type TABLET_DATA_DELETED (replaced by tablet 8adc74a357244102801b2aa8d27d9747) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.884791 914 raft_consensus.cc:3058] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.885764 1185 ts_tablet_manager.cc:1905] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.886724 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b550173c58174bedacdcb184e73e8e9c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.887302 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b550173c58174bedacdcb184e73e8e9c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.887653 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e68bdbba23704093a01c0a5b576d3f6d with delete_type TABLET_DATA_DELETED (replaced by tablet d23d6df63c1a4e02a43bccf625e200bb) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.887928 1184 tablet_replica.cc:331] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.888101 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 with delete_type TABLET_DATA_DELETED (replaced by tablet 8adc74a357244102801b2aa8d27d9747) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.888149 1184 raft_consensus.cc:2241] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.888320 1184 raft_consensus.cc:2270] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.888608 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_16b3928f208649fa879486e9d7d6a235"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.889114 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_16b3928f208649fa879486e9d7d6a235 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.889505 1184 ts_tablet_manager.cc:1905] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.890995 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b4ad86cd8af24e86b952d94a60b5fa41"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.891453 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f8a9399b9f4e464bbcf10515ab300c15"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.891956 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f8a9399b9f4e464bbcf10515ab300c15 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.892086 1185 ts_tablet_manager.cc:1918] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:14.892213 1185 log.cc:1199] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/92ea9fe1c36c4807ba02f23e6877198c
W20250905 19:52:14.891484 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b4ad86cd8af24e86b952d94a60b5fa41 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.892678 1185 ts_tablet_manager.cc:1939] T 92ea9fe1c36c4807ba02f23e6877198c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:14.893718 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e68bdbba23704093a01c0a5b576d3f6d with delete_type TABLET_DATA_DELETED (replaced by tablet d23d6df63c1a4e02a43bccf625e200bb) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.893873 753 tablet_service.cc:1515] Processing DeleteTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 with delete_type TABLET_DATA_DELETED (replaced by tablet 8adc74a357244102801b2aa8d27d9747) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.894389 1171 tablet_bootstrap.cc:492] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:14.894543 1171 ts_tablet_manager.cc:1397] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.056s user 0.008s sys 0.020s
I20250905 19:52:14.894738 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 92ea9fe1c36c4807ba02f23e6877198c (table default.Table_b265ba1d975f43fe9d26542d0296d10d [id=95b523055663453a83fe91e35e1205cc]) successfully deleted
W20250905 19:52:14.895457 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 7e62dbe9e48544109446323b2cc4501f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7e62dbe9e48544109446323b2cc4501f already in progress: creating tablet
I20250905 19:52:14.895463 1171 raft_consensus.cc:357] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.895650 1171 raft_consensus.cc:383] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:14.895749 1171 raft_consensus.cc:738] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:14.896067 1171 consensus_queue.cc:260] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:14.896140 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet e68bdbba23704093a01c0a5b576d3f6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68bdbba23704093a01c0a5b576d3f6d already in progress: creating tablet
I20250905 19:52:14.896291 914 raft_consensus.cc:1273] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:14.898449 1175 consensus_queue.cc:1035] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:14.899271 1171 ts_tablet_manager.cc:1428] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:14.900452 1171 tablet_bootstrap.cc:492] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:14.902220 1184 ts_tablet_manager.cc:1918] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.902377 1171 tablet_bootstrap.cc:654] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:14.902586 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ef9b3f29d521431f9eb8ea06ece31552 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef9b3f29d521431f9eb8ea06ece31552 already in progress: creating tablet
I20250905 19:52:14.902670 1184 log.cc:1199] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e68bdbba23704093a01c0a5b576d3f6d
I20250905 19:52:14.903216 1170 consensus_queue.cc:260] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.903717 1184 ts_tablet_manager.cc:1939] T e68bdbba23704093a01c0a5b576d3f6d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:14.904798 1170 ts_tablet_manager.cc:1428] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.040s user 0.001s sys 0.001s
I20250905 19:52:14.905129 1170 tablet_bootstrap.cc:492] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:14.906955 1170 tablet_bootstrap.cc:654] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:14.907338 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e68bdbba23704093a01c0a5b576d3f6d (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]) successfully deleted
W20250905 19:52:14.908548 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ef9b3f29d521431f9eb8ea06ece31552 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef9b3f29d521431f9eb8ea06ece31552 already in progress: creating tablet
W20250905 19:52:14.910609 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e68bdbba23704093a01c0a5b576d3f6d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e68bdbba23704093a01c0a5b576d3f6d already in progress: creating tablet
W20250905 19:52:14.910945 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ef9b3f29d521431f9eb8ea06ece31552 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ef9b3f29d521431f9eb8ea06ece31552 already in progress: creating tablet
I20250905 19:52:14.914783 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_01b46213458147cd9bf884185bcff837" } new_table_name: "default.Table_edca9fb5f8fb48d28540ecd7e385178c" modify_external_catalogs: true
I20250905 19:52:14.916751 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c865b4a070ba494fab94d0daa544d56f" } new_table_name: "default.Table_db4bfb066c024a73b14c6c9d1f02a6cd" modify_external_catalogs: true
I20250905 19:52:14.920760 886 tablet_service.cc:1468] Processing CreateTablet for tablet b2c7906a718c46d9a76587a70d96a1f9 (DEFAULT_TABLE table=default.Table_f681ca53501741449436b37d7ee502c7 [id=99384aac4e624d6db630b249c3ba2543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.921257 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c7906a718c46d9a76587a70d96a1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.922688 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_87bff647cebf4507bb0e6a6caf45f752"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:14.923273 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87bff647cebf4507bb0e6a6caf45f752 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.925050 890 tablet_service.cc:1468] Processing CreateTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 (DEFAULT_TABLE table=default.Table_056e9cf230e44fc5badd0c9b709c0c4f [id=88e8e8eaaefb4045b64fd8ad64f9990c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.925472 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eacedd2e38d64d2cb25b2ebd88f563a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.925360 1030 tablet_service.cc:1468] Processing CreateTablet for tablet b2c7906a718c46d9a76587a70d96a1f9 (DEFAULT_TABLE table=default.Table_f681ca53501741449436b37d7ee502c7 [id=99384aac4e624d6db630b249c3ba2543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.925905 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c7906a718c46d9a76587a70d96a1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.928349 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a (DEFAULT_TABLE table=default.Table_e688c1faf5c84ad1aa446711ae39d273 [id=d9c9bed59ab9499393be43c41657e6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.928387 1028 tablet_service.cc:1468] Processing CreateTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 (DEFAULT_TABLE table=default.Table_056e9cf230e44fc5badd0c9b709c0c4f [id=88e8e8eaaefb4045b64fd8ad64f9990c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.928705 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0480eda7e20c4336bbeacad1e7cdd56a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.928889 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eacedd2e38d64d2cb25b2ebd88f563a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.932119 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 92ea9fe1c36c4807ba02f23e6877198c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:14.932435 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a (DEFAULT_TABLE table=default.Table_e688c1faf5c84ad1aa446711ae39d273 [id=d9c9bed59ab9499393be43c41657e6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.932888 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0480eda7e20c4336bbeacad1e7cdd56a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:14.941617 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 92ea9fe1c36c4807ba02f23e6877198c because the tablet was not found. No further retry: Not found: Tablet not found: 92ea9fe1c36c4807ba02f23e6877198c
I20250905 19:52:14.954691 1176 raft_consensus.cc:491] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:14.954887 1176 raft_consensus.cc:513] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.955247 753 tablet_service.cc:1468] Processing CreateTablet for tablet b2c7906a718c46d9a76587a70d96a1f9 (DEFAULT_TABLE table=default.Table_f681ca53501741449436b37d7ee502c7 [id=99384aac4e624d6db630b249c3ba2543]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.955518 1176 leader_election.cc:290] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:14.955678 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2c7906a718c46d9a76587a70d96a1f9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.956172 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e2630ff3a4848ada70176448de607e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:14.957021 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e2630ff3a4848ada70176448de607e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:14.957212 776 raft_consensus.cc:2466] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
W20250905 19:52:14.957757 986 leader_election.cc:343] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.958118 984 leader_election.cc:304] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:14.958617 1176 raft_consensus.cc:2802] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:14.958745 1176 raft_consensus.cc:491] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:14.958837 1176 raft_consensus.cc:3058] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.960302 752 tablet_service.cc:1468] Processing CreateTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 (DEFAULT_TABLE table=default.Table_056e9cf230e44fc5badd0c9b709c0c4f [id=88e8e8eaaefb4045b64fd8ad64f9990c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.960682 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet eacedd2e38d64d2cb25b2ebd88f563a1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:14.961709 1176 raft_consensus.cc:513] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:14.962810 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e2630ff3a4848ada70176448de607e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
W20250905 19:52:14.963280 986 leader_election.cc:343] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:14.964090 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0bb119db110748f8be970eaa3c458a06"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:14.964274 756 tablet_service.cc:1468] Processing CreateTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a (DEFAULT_TABLE table=default.Table_e688c1faf5c84ad1aa446711ae39d273 [id=d9c9bed59ab9499393be43c41657e6e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:14.964428 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e2630ff3a4848ada70176448de607e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:14.964597 776 raft_consensus.cc:3058] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:14.964638 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0480eda7e20c4336bbeacad1e7cdd56a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:14.964704 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0bb119db110748f8be970eaa3c458a06 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:14.966920 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7ea4506d60df44b8b9a7f137911b5570" } new_table_name: "default.Table_de29f307cc654a6880fba89258c177e7" modify_external_catalogs: true
W20250905 19:52:14.968427 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e5a7c8a1fe604706be88d0461b241a31 consensus state has no leader
I20250905 19:52:14.968434 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet e5a7c8a1fe604706be88d0461b241a31
I20250905 19:52:14.969535 455 catalog_manager.cc:6652] ReplaceTablet: tablet e5a7c8a1fe604706be88d0461b241a31 deleted and replaced by tablet 6528897cc4b248898a44aad6a9ccaeae
I20250905 19:52:14.969851 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 1e2630ff3a4848ada70176448de607e2
I20250905 19:52:14.970368 453 catalog_manager.cc:6652] ReplaceTablet: tablet 1e2630ff3a4848ada70176448de607e2 deleted and replaced by tablet d20e269bf2224fe49f701c4777c42632
I20250905 19:52:14.971546 755 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.972397 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e2630ff3a4848ada70176448de607e2 with delete_type TABLET_DATA_DELETED (replaced by tablet d20e269bf2224fe49f701c4777c42632) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:14.972623 1186 tablet_replica.cc:331] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:14.973707 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ec810e5cc1af4b9d9f6bd421072bbc77 consensus state has no leader
I20250905 19:52:14.975014 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e2630ff3a4848ada70176448de607e2 with delete_type TABLET_DATA_DELETED (replaced by tablet d20e269bf2224fe49f701c4777c42632) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:14.975744 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 1e2630ff3a4848ada70176448de607e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1e2630ff3a4848ada70176448de607e2 already in progress: creating tablet
I20250905 19:52:14.968883 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d305bf057f3f4754bc3dab037bd823bd" } new_table_name: "default.Table_36557fc6d91f4afb9816c01bb97c550c" modify_external_catalogs: true
I20250905 19:52:14.976203 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.976475 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 987a6193d2a74f90906a45b237040e39 with delete_type TABLET_DATA_DELETED (replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:14.976845 1184 tablet_replica.cc:331] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:14.977061 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:14.977336 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet c2a054ea3bca416b8fa344a5de28d798 because the tablet was not found. No further retry: Not found: Tablet not found: c2a054ea3bca416b8fa344a5de28d798
I20250905 19:52:14.977586 1184 raft_consensus.cc:2241] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.977766 1184 raft_consensus.cc:2270] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:14.977777 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet e5a7c8a1fe604706be88d0461b241a31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5a7c8a1fe604706be88d0461b241a31 already in progress: creating tablet
I20250905 19:52:14.983392 776 raft_consensus.cc:2466] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:14.983633 1186 raft_consensus.cc:2241] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:14.984016 1186 raft_consensus.cc:2270] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:14.984031 984 leader_election.cc:304] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
W20250905 19:52:14.984378 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e5a7c8a1fe604706be88d0461b241a31 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e5a7c8a1fe604706be88d0461b241a31 already in progress: creating tablet
I20250905 19:52:14.984982 1186 ts_tablet_manager.cc:1905] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:14.987537 1176 leader_election.cc:290] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:14.987826 1176 raft_consensus.cc:2802] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:14.989218 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b550173c58174bedacdcb184e73e8e9c" } modify_external_catalogs: true
I20250905 19:52:14.989277 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c32f9552869b4ea686b824e1967c932c" } new_table_name: "default.Table_3c27b9d996a542a1bed7229a301e6a9e" modify_external_catalogs: true
I20250905 19:52:14.989446 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b550173c58174bedacdcb184e73e8e9c" } modify_external_catalogs: true
I20250905 19:52:14.990676 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f8a9399b9f4e464bbcf10515ab300c15" } modify_external_catalogs: true
I20250905 19:52:14.990885 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f8a9399b9f4e464bbcf10515ab300c15" } modify_external_catalogs: true
I20250905 19:52:14.992414 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_db4bfb066c024a73b14c6c9d1f02a6cd" } modify_external_catalogs: true
I20250905 19:52:14.992614 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_db4bfb066c024a73b14c6c9d1f02a6cd" } modify_external_catalogs: true
I20250905 19:52:14.993687 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_056e9cf230e44fc5badd0c9b709c0c4f" } modify_external_catalogs: true
I20250905 19:52:14.993733 1186 ts_tablet_manager.cc:1918] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:14.993835 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_056e9cf230e44fc5badd0c9b709c0c4f" } modify_external_catalogs: true
I20250905 19:52:14.994035 1186 log.cc:1199] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/1e2630ff3a4848ada70176448de607e2
I20250905 19:52:14.994478 1186 ts_tablet_manager.cc:1939] T 1e2630ff3a4848ada70176448de607e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:14.990907 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1eaa6c2091ee4d949f145d0e9e806550" } modify_external_catalogs: true
I20250905 19:52:14.996708 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1eaa6c2091ee4d949f145d0e9e806550" } modify_external_catalogs: true
I20250905 19:52:14.997891 1176 raft_consensus.cc:695] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:14.998124 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 1e2630ff3a4848ada70176448de607e2 (table default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]) successfully deleted
I20250905 19:52:14.998324 1176 consensus_queue.cc:237] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.004321 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 4 got 0
I20250905 19:52:15.004472 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e68bdbba23704093a01c0a5b576d3f6d with delete_type TABLET_DATA_DELETED (replaced by tablet d23d6df63c1a4e02a43bccf625e200bb) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.004794 1184 ts_tablet_manager.cc:1905] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.004827 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e2630ff3a4848ada70176448de607e2 with delete_type TABLET_DATA_DELETED (replaced by tablet d20e269bf2224fe49f701c4777c42632) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.004973 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.005961 890 tablet_service.cc:1515] Processing DeleteTablet for tablet c2a054ea3bca416b8fa344a5de28d798 with delete_type TABLET_DATA_DELETED (replaced by tablet ec810e5cc1af4b9d9f6bd421072bbc77) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.006223 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ca53e56561674c8c830a0e28443a19e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.006508 1185 tablet_replica.cc:331] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.006770 1185 raft_consensus.cc:2241] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.006943 1185 raft_consensus.cc:2270] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.007489 1175 raft_consensus.cc:491] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.007673 1175 raft_consensus.cc:513] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.008421 1175 leader_election.cc:290] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.009604 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266ffb862d144368994f5c33e684647d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:15.010102 1185 ts_tablet_manager.cc:1905] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.011410 1184 ts_tablet_manager.cc:1918] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.011552 1184 log.cc:1199] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/987a6193d2a74f90906a45b237040e39
W20250905 19:52:15.012117 712 leader_election.cc:343] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.018112 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266ffb862d144368994f5c33e684647d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:15.018184 1185 ts_tablet_manager.cc:1918] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.018347 1052 raft_consensus.cc:2466] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:15.018427 1185 log.cc:1199] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/c2a054ea3bca416b8fa344a5de28d798
I20250905 19:52:15.018944 1185 ts_tablet_manager.cc:1939] T c2a054ea3bca416b8fa344a5de28d798 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.019187 1184 ts_tablet_manager.cc:1939] T 987a6193d2a74f90906a45b237040e39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:15.020432 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 987a6193d2a74f90906a45b237040e39 (table default.Table_d305bf057f3f4754bc3dab037bd823bd [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:15.021361 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet c2a054ea3bca416b8fa344a5de28d798 (table default.Table_edca9fb5f8fb48d28540ecd7e385178c [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:15.021765 1185 tablet_replica.cc:331] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.021994 1185 raft_consensus.cc:2241] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:15.022163 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet e68bdbba23704093a01c0a5b576d3f6d because the tablet was not found. No further retry: Not found: Tablet not found: e68bdbba23704093a01c0a5b576d3f6d
I20250905 19:52:15.022243 1185 raft_consensus.cc:2270] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.022784 1184 tablet_replica.cc:331] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:15.023021 1184 raft_consensus.cc:2241] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:15.023476 1184 pending_rounds.cc:70] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:15.023672 1184 pending_rounds.cc:77] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089525761646592 op_type: NO_OP noop_request { }
I20250905 19:52:15.023877 1184 raft_consensus.cc:2887] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:15.024041 1184 raft_consensus.cc:2270] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.024236 1185 ts_tablet_manager.cc:1905] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.025404 1232 mvcc.cc:204] Tried to move back new op lower bound from 7197089525212925952 to 7197089524178034688. Current Snapshot: MvccSnapshot[applied={T|T < 7197089525212925952}]
W20250905 19:52:15.026051 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet afb404e5b06c452bb98abeea95070a5f consensus state has no leader
I20250905 19:52:15.026795 451 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 6528897cc4b248898a44aad6a9ccaeae
I20250905 19:52:15.027478 448 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 6632fc37bcb2466abc24161c521e6d03
I20250905 19:52:15.033529 1184 ts_tablet_manager.cc:1905] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.036427 1185 ts_tablet_manager.cc:1918] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.036620 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 4 got 0
I20250905 19:52:15.036684 1185 log.cc:1199] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ca53e56561674c8c830a0e28443a19e0
I20250905 19:52:15.036875 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
I20250905 19:52:15.037277 1185 ts_tablet_manager.cc:1939] T ca53e56561674c8c830a0e28443a19e0 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.038338 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ca53e56561674c8c830a0e28443a19e0 (table default.Table_0a4b61de931546689e038111b0a405a4 [id=b72b466b7abe468dbaa985c9f469cbcf]) successfully deleted
I20250905 19:52:15.039528 446 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
I20250905 19:52:15.039783 446 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:15.040474 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4f1d960461ef4829960baa5cca7f7765
W20250905 19:52:15.040920 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f842c6ac96ec44ffaae02edb4a9fdeef consensus state has no leader
I20250905 19:52:15.041687 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:15.042824 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because the tablet was not found. No further retry: Not found: Tablet not found: 4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:15.046530 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.046739 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet eacedd2e38d64d2cb25b2ebd88f563a1
I20250905 19:52:15.047140 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:15.047493 1169 tablet_bootstrap.cc:492] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.047641 1169 ts_tablet_manager.cc:1397] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.169s user 0.009s sys 0.002s
I20250905 19:52:15.048323 893 tablet_service.cc:1515] Processing DeleteTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.048378 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.048588 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.049408 651 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:15.050197 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:15.050781 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet eacedd2e38d64d2cb25b2ebd88f563a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eacedd2e38d64d2cb25b2ebd88f563a1 already in progress: creating tablet
W20250905 19:52:15.051095 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6d09371944644b3082268b2b37ce54f8 because the tablet was not found. No further retry: Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
W20250905 19:52:15.051326 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because the tablet was not found. No further retry: Not found: Tablet not found: 4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:15.052515 651 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:15.053324 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:15.054005 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6d09371944644b3082268b2b37ce54f8 because the tablet was not found. No further retry: Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:15.054946 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4f1d960461ef4829960baa5cca7f7765 (DEFAULT_TABLE table=default.Table_b550173c58174bedacdcb184e73e8e9c [id=c56b6d4d3b204b0981bb9cecd8a1a08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.055524 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d960461ef4829960baa5cca7f7765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.055817 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4f1d960461ef4829960baa5cca7f7765 (DEFAULT_TABLE table=default.Table_b550173c58174bedacdcb184e73e8e9c [id=c56b6d4d3b204b0981bb9cecd8a1a08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.056051 893 tablet_service.cc:1468] Processing CreateTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 (DEFAULT_TABLE table=default.Table_dc1ad5f85ed74d81a6bd534a572dc98b [id=5ac15645cb4d4730a4b4464a85299bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.056416 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ac5c7e9b6741d79db2e606d55a3509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.056371 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.056594 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d960461ef4829960baa5cca7f7765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.063062 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.063274 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.063584 1184 ts_tablet_manager.cc:1918] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:15.063732 1184 log.cc:1199] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/1e2630ff3a4848ada70176448de607e2
I20250905 19:52:15.064777 1169 raft_consensus.cc:357] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.065009 1169 raft_consensus.cc:383] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.065127 1169 raft_consensus.cc:738] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.065454 1169 consensus_queue.cc:260] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.066071 1169 ts_tablet_manager.cc:1428] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:15.066505 1169 tablet_bootstrap.cc:492] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.066716 1184 ts_tablet_manager.cc:1939] T 1e2630ff3a4848ada70176448de607e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:15.067822 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 1e2630ff3a4848ada70176448de607e2 (table default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]) successfully deleted
I20250905 19:52:15.069041 711 leader_election.cc:304] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
W20250905 19:52:15.069453 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 50713.5us
I20250905 19:52:15.070446 1203 raft_consensus.cc:2802] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:15.070544 1203 raft_consensus.cc:491] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:15.070652 1203 raft_consensus.cc:3058] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.073506 454 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:15.073993 1203 raft_consensus.cc:513] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.076982 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.078392 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e2630ff3a4848ada70176448de607e2 with delete_type TABLET_DATA_DELETED (replaced by tablet d20e269bf2224fe49f701c4777c42632) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.078701 752 tablet_service.cc:1515] Processing DeleteTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.078863 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.078939 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.079157 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.079411 1203 leader_election.cc:290] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.079766 750 tablet_service.cc:1468] Processing CreateTablet for tablet 4f1d960461ef4829960baa5cca7f7765 (DEFAULT_TABLE table=default.Table_b550173c58174bedacdcb184e73e8e9c [id=c56b6d4d3b204b0981bb9cecd8a1a08c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.084532 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4f1d960461ef4829960baa5cca7f7765. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.085263 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_784717e4261b4c0da57fc511e29222de"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.085956 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_784717e4261b4c0da57fc511e29222de in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.079169 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 987a6193d2a74f90906a45b237040e39 with delete_type TABLET_DATA_DELETED (replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.083714 1175 raft_consensus.cc:491] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.088842 1175 raft_consensus.cc:513] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.089721 1175 leader_election.cc:290] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.090340 1186 tablet_replica.cc:331] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:15.090619 1186 raft_consensus.cc:2241] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.090862 1186 raft_consensus.cc:2270] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.091681 1186 ts_tablet_manager.cc:1905] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.092597 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c9d143854de148aea480b85f2718ef23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:15.092840 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e68bdbba23704093a01c0a5b576d3f6d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:15.093430 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c9d143854de148aea480b85f2718ef23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.093482 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_60f4c59edd6e4f6aa0954a178bcaea7e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.093979 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_60f4c59edd6e4f6aa0954a178bcaea7e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:15.095361 712 leader_election.cc:343] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.096735 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_eb7fe5cbd839447e84d08c2f04f4b051"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.097398 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eb7fe5cbd839447e84d08c2f04f4b051 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.098068 1186 ts_tablet_manager.cc:1918] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.098178 1186 log.cc:1199] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/987a6193d2a74f90906a45b237040e39
I20250905 19:52:15.098557 1186 ts_tablet_manager.cc:1939] T 987a6193d2a74f90906a45b237040e39 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:15.108845 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because the tablet was not found. No further retry: Not found: Tablet not found: 4f1d960461ef4829960baa5cca7f7765
W20250905 19:52:15.109479 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 1e2630ff3a4848ada70176448de607e2 because the tablet was not found. No further retry: Not found: Tablet not found: 1e2630ff3a4848ada70176448de607e2
I20250905 19:52:15.109728 447 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
W20250905 19:52:15.110001 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet eacedd2e38d64d2cb25b2ebd88f563a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eacedd2e38d64d2cb25b2ebd88f563a1 already in progress: creating tablet
W20250905 19:52:15.110337 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because the tablet was not found. No further retry: Not found: Tablet not found: 4f1d960461ef4829960baa5cca7f7765
W20250905 19:52:15.110626 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6d09371944644b3082268b2b37ce54f8 because the tablet was not found. No further retry: Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
W20250905 19:52:15.110944 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6d09371944644b3082268b2b37ce54f8 because the tablet was not found. No further retry: Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:15.111235 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 987a6193d2a74f90906a45b237040e39 (table default.Table_36557fc6d91f4afb9816c01bb97c550c [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:15.112195 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266ffb862d144368994f5c33e684647d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:15.114002 756 tablet_service.cc:1515] Processing DeleteTablet for tablet e68bdbba23704093a01c0a5b576d3f6d with delete_type TABLET_DATA_DELETED (replaced by tablet d23d6df63c1a4e02a43bccf625e200bb) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.114300 1186 tablet_replica.cc:331] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:15.114512 1186 raft_consensus.cc:2241] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.114692 1186 raft_consensus.cc:2270] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.115511 1186 ts_tablet_manager.cc:1905] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:15.115712 712 leader_election.cc:343] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.120713 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1acbec020bbe40ef9e171c59b3b82526"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:15.121176 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_fb7741d7b2074b59ab84f910b06bc8c5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.121351 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1acbec020bbe40ef9e171c59b3b82526 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:15.121675 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb7741d7b2074b59ab84f910b06bc8c5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.121258 1238 tablet.cc:1722] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:15.126989 1186 ts_tablet_manager.cc:1918] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.127179 1186 log.cc:1199] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e68bdbba23704093a01c0a5b576d3f6d
I20250905 19:52:15.127585 1186 ts_tablet_manager.cc:1939] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:15.127925 1239 tablet.cc:1722] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:15.129244 1170 tablet_bootstrap.cc:492] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.129467 1170 ts_tablet_manager.cc:1397] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.224s user 0.014s sys 0.034s
I20250905 19:52:15.130586 1170 raft_consensus.cc:357] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.130810 1170 raft_consensus.cc:383] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.130910 1170 raft_consensus.cc:738] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.131224 1170 consensus_queue.cc:260] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.131848 1170 ts_tablet_manager.cc:1428] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.133241 1170 tablet_bootstrap.cc:492] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.133548 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 987a6193d2a74f90906a45b237040e39 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.135490 1170 tablet_bootstrap.cc:654] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.138350 453 master_service.cc:948] ReplaceTablet: received request to replace tablet d1f668b9ae964b7ebf87726f734e3a40 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.138927 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet e68bdbba23704093a01c0a5b576d3f6d (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]) successfully deleted
I20250905 19:52:15.150488 1171 tablet_bootstrap.cc:492] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.150696 1171 ts_tablet_manager.cc:1397] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.250s user 0.076s sys 0.066s
I20250905 19:52:15.151527 1171 raft_consensus.cc:357] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.151739 1171 raft_consensus.cc:383] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.151823 1171 raft_consensus.cc:738] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.152139 1171 consensus_queue.cc:260] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.152638 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 (DEFAULT_TABLE table=default.Table_7dd83ce427b9438990eee889da0bcb0f [id=8904fcfbf1a94032b62d596e1ea14569]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.153254 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4ec474911941b2a2d0825cdb37bd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.153607 1171 ts_tablet_manager.cc:1428] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:15.154816 1171 tablet_bootstrap.cc:492] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.157554 750 tablet_service.cc:1468] Processing CreateTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 (DEFAULT_TABLE table=default.Table_dc1ad5f85ed74d81a6bd534a572dc98b [id=5ac15645cb4d4730a4b4464a85299bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.158239 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ac5c7e9b6741d79db2e606d55a3509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.158334 1161 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:15.158541 1161 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:15.163589 1171 tablet_bootstrap.cc:654] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.164649 756 tablet_service.cc:1468] Processing CreateTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 (DEFAULT_TABLE table=default.Table_7dd83ce427b9438990eee889da0bcb0f [id=8904fcfbf1a94032b62d596e1ea14569]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.165033 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4ec474911941b2a2d0825cdb37bd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.165792 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 (DEFAULT_TABLE table=default.Table_dc1ad5f85ed74d81a6bd534a572dc98b [id=5ac15645cb4d4730a4b4464a85299bed]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:15.167147 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1d960461ef4829960baa5cca7f7765 already in progress: creating tablet
W20250905 19:52:15.167425 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6d09371944644b3082268b2b37ce54f8 because the tablet was not found. No further retry: Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:15.167716 886 tablet_service.cc:1468] Processing CreateTablet for tablet 6d09371944644b3082268b2b37ce54f8 (DEFAULT_TABLE table=default.Table_1eaa6c2091ee4d949f145d0e9e806550 [id=43eb792904d2448da31df6a56c647397]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.167725 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266ffb862d144368994f5c33e684647d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:15.168233 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b776063b81c44217b517a619af927c08 (DEFAULT_TABLE table=default.Table_ce8e455023774d02ba896de13623c0ea [id=06e2f51841544340adc3a39ca41a9bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.168376 893 tablet_service.cc:1468] Processing CreateTablet for tablet b776063b81c44217b517a619af927c08 (DEFAULT_TABLE table=default.Table_ce8e455023774d02ba896de13623c0ea [id=06e2f51841544340adc3a39ca41a9bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.168538 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 (DEFAULT_TABLE table=default.Table_7dd83ce427b9438990eee889da0bcb0f [id=8904fcfbf1a94032b62d596e1ea14569]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.168805 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_55ca8eccc0224578835d4f9f8fae2311"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:15.167934 1053 raft_consensus.cc:3058] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:15.169155 711 leader_election.cc:343] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: e68bdbba23704093a01c0a5b576d3f6d
W20250905 19:52:15.169370 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_55ca8eccc0224578835d4f9f8fae2311 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.169320 711 leader_election.cc:304] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:15.169586 711 raft_consensus.cc:2684] T e68bdbba23704093a01c0a5b576d3f6d P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:15.171546 887 tablet_service.cc:1468] Processing CreateTablet for tablet 7aa06165136a4fa5a935dfdd786a75c4 (DEFAULT_TABLE table=default.Table_16b3928f208649fa879486e9d7d6a235 [id=ab02be958b034155ba9d2a431944a420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.172477 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78ac5c7e9b6741d79db2e606d55a3509. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.175544 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e688c1faf5c84ad1aa446711ae39d273" } new_table_name: "default.Table_6c729e17dd76402eb3cd6ed85f3d9875" modify_external_catalogs: true
I20250905 19:52:15.176101 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9554a085b43c42a98edcb684be28faaa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.176786 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9554a085b43c42a98edcb684be28faaa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.177546 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b776063b81c44217b517a619af927c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.178153 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_5996e867324248b1b6c68868f6dfe8e2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.178638 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5996e867324248b1b6c68868f6dfe8e2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.179888 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e7d692a2d299432090b66b38168b5894" } new_table_name: "default.Table_44155b42e0d04fc0b3a53308c495f845" modify_external_catalogs: true
I20250905 19:52:15.184072 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f4ec474911941b2a2d0825cdb37bd72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.185653 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e5751a29750a47e8b4b75af6d76edd5d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.186210 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5751a29750a47e8b4b75af6d76edd5d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:15.186725 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet eacedd2e38d64d2cb25b2ebd88f563a1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet eacedd2e38d64d2cb25b2ebd88f563a1 already in progress: creating tablet
I20250905 19:52:15.187402 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1acbec020bbe40ef9e171c59b3b82526" } new_table_name: "default.Table_89a831c4cad3457b86b01609bfa3a57b" modify_external_catalogs: true
W20250905 19:52:15.189343 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4f1d960461ef4829960baa5cca7f7765 already in progress: creating tablet
W20250905 19:52:15.189589 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6d09371944644b3082268b2b37ce54f8 because the tablet was not found. No further retry: Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:15.189788 1053 raft_consensus.cc:2466] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:15.190847 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 6d09371944644b3082268b2b37ce54f8 (DEFAULT_TABLE table=default.Table_1eaa6c2091ee4d949f145d0e9e806550 [id=43eb792904d2448da31df6a56c647397]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:15.191087 440 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet f842c6ac96ec44ffaae02edb4a9fdeef (table default.Table_36557fc6d91f4afb9816c01bb97c550c [id=7a9f1417bfd04c158dbedc9ea02d6dce], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet f842c6ac96ec44ffaae02edb4a9fdeef consensus state has no leader
I20250905 19:52:15.191440 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d09371944644b3082268b2b37ce54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.192750 756 tablet_service.cc:1468] Processing CreateTablet for tablet b776063b81c44217b517a619af927c08 (DEFAULT_TABLE table=default.Table_ce8e455023774d02ba896de13623c0ea [id=06e2f51841544340adc3a39ca41a9bea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.193535 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b776063b81c44217b517a619af927c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.193915 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 7aa06165136a4fa5a935dfdd786a75c4 (DEFAULT_TABLE table=default.Table_16b3928f208649fa879486e9d7d6a235 [id=ab02be958b034155ba9d2a431944a420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.194324 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa06165136a4fa5a935dfdd786a75c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.195762 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b776063b81c44217b517a619af927c08. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.196468 711 leader_election.cc:304] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:15.197217 1196 raft_consensus.cc:2802] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:15.198081 754 tablet_service.cc:1468] Processing CreateTablet for tablet 6d09371944644b3082268b2b37ce54f8 (DEFAULT_TABLE table=default.Table_1eaa6c2091ee4d949f145d0e9e806550 [id=43eb792904d2448da31df6a56c647397]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.198076 1196 raft_consensus.cc:695] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:15.199155 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d09371944644b3082268b2b37ce54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.201665 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d09371944644b3082268b2b37ce54f8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.201889 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa06165136a4fa5a935dfdd786a75c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.203336 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7aa06165136a4fa5a935dfdd786a75c4 (DEFAULT_TABLE table=default.Table_16b3928f208649fa879486e9d7d6a235 [id=ab02be958b034155ba9d2a431944a420]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.203891 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7aa06165136a4fa5a935dfdd786a75c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.204564 1196 consensus_queue.cc:237] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.218926 1203 raft_consensus.cc:491] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.219182 1203 raft_consensus.cc:513] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.219486 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.220006 1203 leader_election.cc:290] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.220091 1169 tablet_bootstrap.cc:654] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.222473 447 master_service.cc:948] ReplaceTablet: received request to replace tablet f842c6ac96ec44ffaae02edb4a9fdeef from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.224104 453 catalog_manager.cc:6652] ReplaceTablet: tablet d1f668b9ae964b7ebf87726f734e3a40 deleted and replaced by tablet be73873da994402e84fff6aa2ded7525
I20250905 19:52:15.224234 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffa14750f1b94c589779f7da0e764ccc" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:15.224339 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 with delete_type TABLET_DATA_DELETED (replaced by tablet be73873da994402e84fff6aa2ded7525) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.224512 1053 raft_consensus.cc:2466] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:15.224589 1185 tablet_replica.cc:331] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.224787 1185 raft_consensus.cc:2241] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.224973 1185 raft_consensus.cc:2270] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.225289 711 leader_election.cc:304] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:15.225668 1203 raft_consensus.cc:2802] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:15.225777 1203 raft_consensus.cc:491] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:15.225852 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e2630ff3a4848ada70176448de607e2 with delete_type TABLET_DATA_DELETED (replaced by tablet d20e269bf2224fe49f701c4777c42632) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.225886 1203 raft_consensus.cc:3058] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.225989 453 master_service.cc:948] ReplaceTablet: received request to replace tablet f842c6ac96ec44ffaae02edb4a9fdeef from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.226601 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 with delete_type TABLET_DATA_DELETED (replaced by tablet be73873da994402e84fff6aa2ded7525) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.226867 1184 tablet_replica.cc:331] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:15.229514 1184 raft_consensus.cc:2241] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.230265 1184 raft_consensus.cc:2270] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.230643 1203 raft_consensus.cc:513] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.231393 1240 tablet.cc:1722] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:15.232329 1169 tablet_bootstrap.cc:492] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.232360 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffa14750f1b94c589779f7da0e764ccc" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:15.232509 1169 ts_tablet_manager.cc:1397] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.166s user 0.004s sys 0.000s
W20250905 19:52:15.232892 712 leader_election.cc:343] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.233361 1169 raft_consensus.cc:357] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.233556 1169 raft_consensus.cc:383] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.233631 1169 raft_consensus.cc:738] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.233888 1203 leader_election.cc:290] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.234570 1171 tablet_bootstrap.cc:492] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.234748 1171 ts_tablet_manager.cc:1397] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.080s user 0.005s sys 0.009s
I20250905 19:52:15.234730 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffa14750f1b94c589779f7da0e764ccc" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:15.235064 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ffa14750f1b94c589779f7da0e764ccc" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
W20250905 19:52:15.235327 712 leader_election.cc:343] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.235225 1052 raft_consensus.cc:3058] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.235617 1171 raft_consensus.cc:357] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.235777 1171 raft_consensus.cc:383] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.235877 1171 raft_consensus.cc:738] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.236299 1171 consensus_queue.cc:260] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.236528 1180 raft_consensus.cc:491] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.236660 1180 raft_consensus.cc:513] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.237278 1180 leader_election.cc:290] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.238250 754 tablet_service.cc:1515] Processing DeleteTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 with delete_type TABLET_DATA_DELETED (replaced by tablet be73873da994402e84fff6aa2ded7525) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.238539 1186 tablet_replica.cc:331] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:15.238729 1186 raft_consensus.cc:2241] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:15.239022 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266ffb862d144368994f5c33e684647d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.239156 1186 raft_consensus.cc:2270] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.239447 1052 raft_consensus.cc:2466] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:15.239638 1171 ts_tablet_manager.cc:1428] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:15.239696 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "266ffb862d144368994f5c33e684647d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:15.239946 1053 raft_consensus.cc:2391] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 835daf30e4324782848290be181ae78f in current term 1: Already voted for candidate f21c498a0d964e8fbebfd1b3cb1916fe in this term.
I20250905 19:52:15.240144 1171 tablet_bootstrap.cc:492] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.242339 1171 tablet_bootstrap.cc:654] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.233870 1169 consensus_queue.cc:260] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.243638 1169 ts_tablet_manager.cc:1428] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:15.244422 1169 tablet_bootstrap.cc:492] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.246085 1180 raft_consensus.cc:491] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.246356 1169 tablet_bootstrap.cc:654] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.246263 1180 raft_consensus.cc:513] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.246883 1180 leader_election.cc:290] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.247727 1184 ts_tablet_manager.cc:1905] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.248319 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
I20250905 19:52:15.248922 448 master_service.cc:948] ReplaceTablet: received request to replace tablet ef9b3f29d521431f9eb8ea06ece31552 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.249423 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8cdd361d6e274d5eb920e324573d83fd" } new_table_name: "default.Table_eb3258db08db42738115c037a4d2912d" modify_external_catalogs: true
W20250905 19:52:15.251837 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbab0209a48d4ad8b2f2dec0562e9598 has no consensus state
I20250905 19:52:15.252293 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0bb119db110748f8be970eaa3c458a06" } modify_external_catalogs: true
I20250905 19:52:15.252456 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0bb119db110748f8be970eaa3c458a06" } modify_external_catalogs: true
I20250905 19:52:15.252761 711 leader_election.cc:304] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:15.253473 848 leader_election.cc:304] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:15.253809 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_784717e4261b4c0da57fc511e29222de" } modify_external_catalogs: true
I20250905 19:52:15.253949 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_784717e4261b4c0da57fc511e29222de" } modify_external_catalogs: true
W20250905 19:52:15.254021 848 leader_election.cc:343] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 1e2630ff3a4848ada70176448de607e2
I20250905 19:52:15.254263 1180 raft_consensus.cc:3058] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.254853 1184 ts_tablet_manager.cc:1918] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:15.254979 1184 log.cc:1199] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.255514 1184 ts_tablet_manager.cc:1939] T d1f668b9ae964b7ebf87726f734e3a40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:15.257241 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ffa14750f1b94c589779f7da0e764ccc consensus state has no leader
I20250905 19:52:15.257774 1180 raft_consensus.cc:2747] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:15.258618 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0480eda7e20c4336bbeacad1e7cdd56a consensus state has no leader
W20250905 19:52:15.258972 849 leader_election.cc:343] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 1e2630ff3a4848ada70176448de607e2
I20250905 19:52:15.259155 849 leader_election.cc:304] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:15.259614 1180 raft_consensus.cc:2747] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:15.259693 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]) successfully deleted
I20250905 19:52:15.259924 1196 raft_consensus.cc:2802] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:15.260140 1196 raft_consensus.cc:695] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:15.260485 1196 consensus_queue.cc:237] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.261049 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7dd83ce427b9438990eee889da0bcb0f" } modify_external_catalogs: true
I20250905 19:52:15.261219 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7dd83ce427b9438990eee889da0bcb0f" } modify_external_catalogs: true
I20250905 19:52:15.263397 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3c27b9d996a542a1bed7229a301e6a9e" } modify_external_catalogs: true
I20250905 19:52:15.263650 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3c27b9d996a542a1bed7229a301e6a9e" } modify_external_catalogs: true
I20250905 19:52:15.265748 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_dc1ad5f85ed74d81a6bd534a572dc98b" } new_table_name: "default.Table_755dce6406b647a1b2fffb9aa254e403" modify_external_catalogs: true
I20250905 19:52:15.265745 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d23d6df63c1a4e02a43bccf625e200bb (DEFAULT_TABLE table=default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.266201 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d23d6df63c1a4e02a43bccf625e200bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.266350 893 tablet_service.cc:1468] Processing CreateTablet for tablet 8adc74a357244102801b2aa8d27d9747 (DEFAULT_TABLE table=default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.266757 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adc74a357244102801b2aa8d27d9747. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.267146 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fb7741d7b2074b59ab84f910b06bc8c5" } modify_external_catalogs: true
I20250905 19:52:15.267308 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fb7741d7b2074b59ab84f910b06bc8c5" } modify_external_catalogs: true
I20250905 19:52:15.270524 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8adc74a357244102801b2aa8d27d9747 (DEFAULT_TABLE table=default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.270928 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adc74a357244102801b2aa8d27d9747. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:15.272619 1223 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48712: Illegal state: RaftConsensus is not running: State = Stopped
I20250905 19:52:15.273046 887 tablet_service.cc:1468] Processing CreateTablet for tablet d23d6df63c1a4e02a43bccf625e200bb (DEFAULT_TABLE table=default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.273563 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d23d6df63c1a4e02a43bccf625e200bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.275827 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet e25832a05f034c25b20b55adfdcb63d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:15.276461 1223 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48712: Illegal state: RaftConsensus is not running: State = Stopped
I20250905 19:52:15.280498 1184 tablet_replica.cc:331] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:15.281827 1185 ts_tablet_manager.cc:1905] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.282105 1184 raft_consensus.cc:2241] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.282303 1184 raft_consensus.cc:2270] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.282939 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8adc74a357244102801b2aa8d27d9747 (DEFAULT_TABLE table=default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.283083 1184 ts_tablet_manager.cc:1905] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.283391 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8adc74a357244102801b2aa8d27d9747. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:15.288322 1223 tablet_service.cc:696] failed op from {username='slave'} at 127.0.0.1:48712: Illegal state: RaftConsensus is not running: State = Stopped
I20250905 19:52:15.288410 754 tablet_service.cc:1468] Processing CreateTablet for tablet d23d6df63c1a4e02a43bccf625e200bb (DEFAULT_TABLE table=default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.289063 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d23d6df63c1a4e02a43bccf625e200bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.289909 1184 ts_tablet_manager.cc:1918] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.290063 1184 log.cc:1199] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e25832a05f034c25b20b55adfdcb63d7
I20250905 19:52:15.290439 1184 ts_tablet_manager.cc:1939] T e25832a05f034c25b20b55adfdcb63d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:15.291604 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e25832a05f034c25b20b55adfdcb63d7 (table default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 [id=f483c42f9aea43f38894538f57e7db56]) successfully deleted
I20250905 19:52:15.293372 1186 ts_tablet_manager.cc:1905] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:15.293380 440 catalog_manager.cc:5007] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): alter failed for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]): Illegal state: RaftConsensus is not running: State = Stopped
I20250905 19:52:15.296617 1185 ts_tablet_manager.cc:1918] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:15.296849 1185 log.cc:1199] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.297363 1185 ts_tablet_manager.cc:1939] T d1f668b9ae964b7ebf87726f734e3a40 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.297556 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet f842c6ac96ec44ffaae02edb4a9fdeef
I20250905 19:52:15.298060 447 catalog_manager.cc:6652] ReplaceTablet: tablet f842c6ac96ec44ffaae02edb4a9fdeef deleted and replaced by tablet f8d3cbdda0c94e139259d90919dc2f95
I20250905 19:52:15.298575 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]) successfully deleted
I20250905 19:52:15.298844 887 tablet_service.cc:1515] Processing DeleteTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef with delete_type TABLET_DATA_DELETED (replaced by tablet f8d3cbdda0c94e139259d90919dc2f95) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.299458 1170 tablet_bootstrap.cc:492] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.299661 1170 ts_tablet_manager.cc:1397] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.166s user 0.011s sys 0.006s
I20250905 19:52:15.299893 1186 ts_tablet_manager.cc:1918] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:15.300072 1186 log.cc:1199] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.300544 1186 ts_tablet_manager.cc:1939] T d1f668b9ae964b7ebf87726f734e3a40 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:15.300561 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef with delete_type TABLET_DATA_DELETED (replaced by tablet f8d3cbdda0c94e139259d90919dc2f95) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.301534 1185 tablet_replica.cc:331] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:15.301589 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbf0b20630b44230b17a80cdb53322fa consensus state has no leader
I20250905 19:52:15.301738 1185 raft_consensus.cc:2241] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.301964 1185 raft_consensus.cc:2270] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.303005 1170 raft_consensus.cc:357] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.303262 1170 raft_consensus.cc:383] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.303362 1170 raft_consensus.cc:738] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.304764 1185 ts_tablet_manager.cc:1905] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:15.305800 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet f842c6ac96ec44ffaae02edb4a9fdeef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f842c6ac96ec44ffaae02edb4a9fdeef already in progress: creating tablet
I20250905 19:52:15.306917 756 tablet_service.cc:1515] Processing DeleteTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef with delete_type TABLET_DATA_DELETED (replaced by tablet f8d3cbdda0c94e139259d90919dc2f95) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.307528 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]) successfully deleted
W20250905 19:52:15.308781 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet f842c6ac96ec44ffaae02edb4a9fdeef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f842c6ac96ec44ffaae02edb4a9fdeef already in progress: creating tablet
I20250905 19:52:15.309069 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 4b27442996a448ac9b9cada3fae2f473
I20250905 19:52:15.309564 1171 tablet_bootstrap.cc:492] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.309725 1171 ts_tablet_manager.cc:1397] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.014s
I20250905 19:52:15.310413 1170 consensus_queue.cc:260] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.310514 1171 raft_consensus.cc:357] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.310688 1171 raft_consensus.cc:383] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.310777 1171 raft_consensus.cc:738] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.311009 1171 consensus_queue.cc:260] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.311487 1171 ts_tablet_manager.cc:1428] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:15.311777 1171 tablet_bootstrap.cc:492] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.312826 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 32913a9596b9440cb920af0d187da1d4
I20250905 19:52:15.313444 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6f4ec474911941b2a2d0825cdb37bd72
I20250905 19:52:15.313553 1171 tablet_bootstrap.cc:654] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.314525 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.314885 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.316664 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet afb404e5b06c452bb98abeea95070a5f
I20250905 19:52:15.316819 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:15.317584 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 78ac5c7e9b6741d79db2e606d55a3509 consensus state has no leader
I20250905 19:52:15.317723 887 tablet_service.cc:1515] Processing DeleteTablet for tablet afb404e5b06c452bb98abeea95070a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.318393 452 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2ec3d150daca48e28677b17322fb4c23
I20250905 19:52:15.318583 1170 ts_tablet_manager.cc:1428] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.019s user 0.002s sys 0.001s
I20250905 19:52:15.319988 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_2ccbbb56da30450a9d81bed4eeadc8ef"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.320533 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2ccbbb56da30450a9d81bed4eeadc8ef in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.322481 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet afb404e5b06c452bb98abeea95070a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:15.325158 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6f4ec474911941b2a2d0825cdb37bd72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f4ec474911941b2a2d0825cdb37bd72 already in progress: creating tablet
I20250905 19:52:15.325165 756 tablet_service.cc:1515] Processing DeleteTablet for tablet afb404e5b06c452bb98abeea95070a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.325604 1170 tablet_bootstrap.cc:492] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.326349 1171 tablet_bootstrap.cc:492] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.326514 1171 ts_tablet_manager.cc:1397] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.004s
I20250905 19:52:15.326857 1169 tablet_bootstrap.cc:492] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.327000 1169 ts_tablet_manager.cc:1397] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.083s user 0.016s sys 0.013s
I20250905 19:52:15.327414 1170 tablet_bootstrap.cc:654] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.327859 1169 raft_consensus.cc:357] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.328099 1169 raft_consensus.cc:383] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.328042 1171 raft_consensus.cc:357] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.328188 1169 raft_consensus.cc:738] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.328229 1171 raft_consensus.cc:383] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.328307 1171 raft_consensus.cc:738] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:15.328526 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet afb404e5b06c452bb98abeea95070a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afb404e5b06c452bb98abeea95070a5f already in progress: creating tablet
W20250905 19:52:15.328917 440 catalog_manager.cc:5001] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): alter failed for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]),no further retry: Not found: Tablet not found: d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.329003 1171 consensus_queue.cc:260] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.329217 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_90d6fa79cea5482eab5d6b62b5d7181b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:15.329267 1169 consensus_queue.cc:260] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:15.329823 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90d6fa79cea5482eab5d6b62b5d7181b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.331133 1169 ts_tablet_manager.cc:1428] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:15.331763 1169 tablet_bootstrap.cc:492] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.332234 1171 ts_tablet_manager.cc:1428] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 19:52:15.333428 1171 tablet_bootstrap.cc:492] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.333550 1169 tablet_bootstrap.cc:654] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.333935 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.334465 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.335417 1171 tablet_bootstrap.cc:654] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.336063 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_164521ea6f9342c8b5380153cb22b888"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.336598 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_164521ea6f9342c8b5380153cb22b888 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:15.339443 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6f4ec474911941b2a2d0825cdb37bd72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f4ec474911941b2a2d0825cdb37bd72 already in progress: creating tablet
I20250905 19:52:15.340301 1185 ts_tablet_manager.cc:1918] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.340435 1185 log.cc:1199] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/1e2630ff3a4848ada70176448de607e2
W20250905 19:52:15.340435 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet afb404e5b06c452bb98abeea95070a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afb404e5b06c452bb98abeea95070a5f already in progress: creating tablet
I20250905 19:52:15.340794 1185 ts_tablet_manager.cc:1939] T 1e2630ff3a4848ada70176448de607e2 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.341771 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 1e2630ff3a4848ada70176448de607e2 (table default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]) successfully deleted
W20250905 19:52:15.344290 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet f842c6ac96ec44ffaae02edb4a9fdeef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f842c6ac96ec44ffaae02edb4a9fdeef already in progress: creating tablet
W20250905 19:52:15.344578 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6f4ec474911941b2a2d0825cdb37bd72 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f4ec474911941b2a2d0825cdb37bd72 already in progress: creating tablet
W20250905 19:52:15.344862 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet afb404e5b06c452bb98abeea95070a5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet afb404e5b06c452bb98abeea95070a5f already in progress: creating tablet
I20250905 19:52:15.346155 1171 tablet_bootstrap.cc:492] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.346153 1169 tablet_bootstrap.cc:492] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.346323 1169 ts_tablet_manager.cc:1397] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.006s
I20250905 19:52:15.346323 1171 ts_tablet_manager.cc:1397] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.000s
I20250905 19:52:15.347071 1171 raft_consensus.cc:357] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.347131 1169 raft_consensus.cc:357] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.347299 1171 raft_consensus.cc:383] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.347312 1169 raft_consensus.cc:383] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.347373 1171 raft_consensus.cc:738] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.347381 1169 raft_consensus.cc:738] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.347657 1169 consensus_queue.cc:260] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.348347 1169 ts_tablet_manager.cc:1428] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.348654 1169 tablet_bootstrap.cc:492] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.347620 1171 consensus_queue.cc:260] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.353628 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 6f4ec474911941b2a2d0825cdb37bd72 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.354912 1169 tablet_bootstrap.cc:654] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.357223 1171 ts_tablet_manager.cc:1428] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:15.360430 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6a14b2be50f942d8bfd8c47247512879"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.361238 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a14b2be50f942d8bfd8c47247512879 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.371205 1171 tablet_bootstrap.cc:492] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.382184 1171 tablet_bootstrap.cc:654] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.389587 1171 tablet_bootstrap.cc:492] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.389830 1171 ts_tablet_manager.cc:1397] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.003s
I20250905 19:52:15.390909 1171 raft_consensus.cc:357] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.391180 1171 raft_consensus.cc:383] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.391290 1171 raft_consensus.cc:738] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.391619 1171 consensus_queue.cc:260] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.396358 1171 ts_tablet_manager.cc:1428] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:15.397282 1171 tablet_bootstrap.cc:492] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.399155 1171 tablet_bootstrap.cc:654] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.401074 1169 tablet_bootstrap.cc:492] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.401237 1169 ts_tablet_manager.cc:1397] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.053s user 0.014s sys 0.032s
I20250905 19:52:15.402020 1169 raft_consensus.cc:357] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.402165 1169 raft_consensus.cc:383] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.402225 1169 raft_consensus.cc:738] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.402420 1169 consensus_queue.cc:260] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.403016 1169 ts_tablet_manager.cc:1428] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.403757 1169 tablet_bootstrap.cc:492] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.405407 1170 tablet_bootstrap.cc:492] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.405476 1169 tablet_bootstrap.cc:654] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.405619 1170 ts_tablet_manager.cc:1397] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.087s user 0.006s sys 0.027s
I20250905 19:52:15.406369 1170 raft_consensus.cc:357] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.406566 1170 raft_consensus.cc:383] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.406656 1170 raft_consensus.cc:738] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.406875 1170 consensus_queue.cc:260] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.407359 1170 ts_tablet_manager.cc:1428] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.407641 1170 tablet_bootstrap.cc:492] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.409052 1171 tablet_bootstrap.cc:492] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.409238 1171 ts_tablet_manager.cc:1397] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.003s
I20250905 19:52:15.409621 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:15.410079 1171 raft_consensus.cc:357] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.411849 1170 tablet_bootstrap.cc:654] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.412212 450 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_edca9fb5f8fb48d28540ecd7e385178c [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 3 got 0
I20250905 19:52:15.413462 455 catalog_manager.cc:5582] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:15.413890 455 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ffa14750f1b94c589779f7da0e764ccc (table default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685]). Expected version 1 got 0
I20250905 19:52:15.414150 455 catalog_manager.cc:5582] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:15.414489 455 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
I20250905 19:52:15.415242 887 tablet_service.cc:1515] Processing DeleteTablet for tablet e68bdbba23704093a01c0a5b576d3f6d with delete_type TABLET_DATA_DELETED (replaced by tablet d23d6df63c1a4e02a43bccf625e200bb) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.415453 450 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_edca9fb5f8fb48d28540ecd7e385178c [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 3 got 0
I20250905 19:52:15.415516 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 987a6193d2a74f90906a45b237040e39 with delete_type TABLET_DATA_DELETED (replaced by tablet f842c6ac96ec44ffaae02edb4a9fdeef) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.415720 1185 tablet_replica.cc:331] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.415954 1185 raft_consensus.cc:2241] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.416203 1185 raft_consensus.cc:2270] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.416211 893 tablet_service.cc:1468] Processing CreateTablet for tablet 813f208881ac4782874138987a710d93 (DEFAULT_TABLE table=default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.416803 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e25832a05f034c25b20b55adfdcb63d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.417061 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813f208881ac4782874138987a710d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.418906 890 tablet_service.cc:1468] Processing CreateTablet for tablet d7d7567807594201a598d715ffbb318b (DEFAULT_TABLE table=default.Table_e5751a29750a47e8b4b75af6d76edd5d [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.419306 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d7567807594201a598d715ffbb318b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.419490 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e62dbe9e48544109446323b2cc4501f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.419734 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef with delete_type TABLET_DATA_DELETED (replaced by tablet f8d3cbdda0c94e139259d90919dc2f95) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.420159 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 813f208881ac4782874138987a710d93 (DEFAULT_TABLE table=default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.420390 887 tablet_service.cc:1468] Processing CreateTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb (DEFAULT_TABLE table=default.Table_5996e867324248b1b6c68868f6dfe8e2 [id=2bf888448df6476f818a8efe70480c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.420527 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813f208881ac4782874138987a710d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.420537 1185 ts_tablet_manager.cc:1905] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.420738 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305a7758527d4d52a669b2f2c4ceedbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.425175 754 tablet_service.cc:1468] Processing CreateTablet for tablet 813f208881ac4782874138987a710d93 (DEFAULT_TABLE table=default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.425700 755 tablet_service.cc:1468] Processing CreateTablet for tablet d7d7567807594201a598d715ffbb318b (DEFAULT_TABLE table=default.Table_e5751a29750a47e8b4b75af6d76edd5d [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.426054 1171 raft_consensus.cc:383] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.426183 1171 raft_consensus.cc:738] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.426249 1176 raft_consensus.cc:491] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.426294 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 813f208881ac4782874138987a710d93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.426409 1176 raft_consensus.cc:513] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.427059 1176 leader_election.cc:290] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:15.427603 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d7567807594201a598d715ffbb318b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.428087 1171 consensus_queue.cc:260] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.428639 887 tablet_service.cc:1468] Processing CreateTablet for tablet fbab0209a48d4ad8b2f2dec0562e9598 (DEFAULT_TABLE table=default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.430434 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbab0209a48d4ad8b2f2dec0562e9598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.431512 1171 ts_tablet_manager.cc:1428] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.022s user 0.001s sys 0.002s
I20250905 19:52:15.432864 756 tablet_service.cc:1468] Processing CreateTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb (DEFAULT_TABLE table=default.Table_5996e867324248b1b6c68868f6dfe8e2 [id=2bf888448df6476f818a8efe70480c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.432960 1184 tablet_replica.cc:331] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:15.433138 1184 raft_consensus.cc:2241] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.433229 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305a7758527d4d52a669b2f2c4ceedbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.433319 1184 raft_consensus.cc:2270] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.434087 1184 ts_tablet_manager.cc:1905] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.434471 1185 ts_tablet_manager.cc:1918] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.434597 1185 log.cc:1199] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e68bdbba23704093a01c0a5b576d3f6d
I20250905 19:52:15.434906 1185 ts_tablet_manager.cc:1939] T e68bdbba23704093a01c0a5b576d3f6d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.435911 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e68bdbba23704093a01c0a5b576d3f6d (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]) successfully deleted
I20250905 19:52:15.436527 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17d0a48c25b84facb082670cb898468c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:15.436774 912 raft_consensus.cc:2466] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:15.437115 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17d0a48c25b84facb082670cb898468c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.437309 776 raft_consensus.cc:2466] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:15.437722 986 leader_election.cc:304] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:15.438077 1176 raft_consensus.cc:2802] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:15.438174 1176 raft_consensus.cc:491] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:15.438284 1176 raft_consensus.cc:3058] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.440027 1171 tablet_bootstrap.cc:492] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.440968 1176 raft_consensus.cc:513] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.441912 1184 ts_tablet_manager.cc:1918] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.441746 1176 leader_election.cc:290] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:15.442040 1184 log.cc:1199] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/7e62dbe9e48544109446323b2cc4501f
I20250905 19:52:15.442368 1184 ts_tablet_manager.cc:1939] T 7e62dbe9e48544109446323b2cc4501f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:15.442353 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17d0a48c25b84facb082670cb898468c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:15.442524 912 raft_consensus.cc:3058] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.444227 755 tablet_service.cc:1468] Processing CreateTablet for tablet fbab0209a48d4ad8b2f2dec0562e9598 (DEFAULT_TABLE table=default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.444660 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbab0209a48d4ad8b2f2dec0562e9598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.445428 912 raft_consensus.cc:2466] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:15.445778 986 leader_election.cc:304] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:15.446075 1176 raft_consensus.cc:2802] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:15.446245 1176 raft_consensus.cc:695] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:15.446496 1176 consensus_queue.cc:237] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.447175 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 7e62dbe9e48544109446323b2cc4501f (table default.Table_3863927d1ffa41289c4ccd14dcec05f9 [id=567a4196a6af4e2b828347357c44313c]) successfully deleted
I20250905 19:52:15.447438 886 tablet_service.cc:1468] Processing CreateTablet for tablet 85c648f799304fa9a94bec665a2dcf97 (DEFAULT_TABLE table=default.Table_eb7fe5cbd839447e84d08c2f04f4b051 [id=7b0316448a484002a5cc2e3db8d4cdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.447818 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c648f799304fa9a94bec665a2dcf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.448210 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 8adc74a357244102801b2aa8d27d9747 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.449786 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb (DEFAULT_TABLE table=default.Table_5996e867324248b1b6c68868f6dfe8e2 [id=2bf888448df6476f818a8efe70480c4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.450176 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 305a7758527d4d52a669b2f2c4ceedbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.452271 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fbab0209a48d4ad8b2f2dec0562e9598 (DEFAULT_TABLE table=default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.452464 891 tablet_service.cc:1468] Processing CreateTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (DEFAULT_TABLE table=default.Table_c9d143854de148aea480b85f2718ef23 [id=6b4e399e326f47ad8c3ca4d5e33538d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.452621 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbab0209a48d4ad8b2f2dec0562e9598. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.452809 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09be30768f5742cf9a4bf1e3fe86dfe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.454341 1184 tablet_replica.cc:331] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:15.455287 1184 raft_consensus.cc:2241] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.455586 1030 tablet_service.cc:1468] Processing CreateTablet for tablet d7d7567807594201a598d715ffbb318b (DEFAULT_TABLE table=default.Table_e5751a29750a47e8b4b75af6d76edd5d [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.456075 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7d7567807594201a598d715ffbb318b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.456362 1184 raft_consensus.cc:2270] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.456549 894 tablet_service.cc:1468] Processing CreateTablet for tablet 1c6295c8c7874df89662b8cef6459fec (DEFAULT_TABLE table=default.Table_87bff647cebf4507bb0e6a6caf45f752 [id=2151f6f3348c4d1fb376448ecfcbae8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.456930 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6295c8c7874df89662b8cef6459fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.457669 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 85c648f799304fa9a94bec665a2dcf97 (DEFAULT_TABLE table=default.Table_eb7fe5cbd839447e84d08c2f04f4b051 [id=7b0316448a484002a5cc2e3db8d4cdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.458182 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c648f799304fa9a94bec665a2dcf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.460170 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "17d0a48c25b84facb082670cb898468c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:15.460301 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (DEFAULT_TABLE table=default.Table_c9d143854de148aea480b85f2718ef23 [id=6b4e399e326f47ad8c3ca4d5e33538d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.460397 776 raft_consensus.cc:3058] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.460391 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 1c6295c8c7874df89662b8cef6459fec (DEFAULT_TABLE table=default.Table_87bff647cebf4507bb0e6a6caf45f752 [id=2151f6f3348c4d1fb376448ecfcbae8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.460682 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09be30768f5742cf9a4bf1e3fe86dfe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.461357 892 tablet_service.cc:1468] Processing CreateTablet for tablet 09b46a12076e492e89d0c0178bcf0524 (DEFAULT_TABLE table=default.Table_60f4c59edd6e4f6aa0954a178bcaea7e [id=3b8967eb30ad46cdbf272136f23318f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.461714 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b46a12076e492e89d0c0178bcf0524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.461723 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_363859b13b284790a71ced21040081e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.462275 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_363859b13b284790a71ced21040081e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.463246 776 raft_consensus.cc:2466] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:15.463598 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ffa14750f1b94c589779f7da0e764ccc (table default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685]). Expected version 1 got 0
I20250905 19:52:15.463608 750 tablet_service.cc:1468] Processing CreateTablet for tablet 85c648f799304fa9a94bec665a2dcf97 (DEFAULT_TABLE table=default.Table_eb7fe5cbd839447e84d08c2f04f4b051 [id=7b0316448a484002a5cc2e3db8d4cdf4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.464063 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85c648f799304fa9a94bec665a2dcf97. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.464262 888 tablet_service.cc:1468] Processing CreateTablet for tablet 9e421213b7bd4dd8a03515519395de7d (DEFAULT_TABLE table=default.Table_b4ad86cd8af24e86b952d94a60b5fa41 [id=152edbecfe3648659c48a2cf89ff9798]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.464643 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e421213b7bd4dd8a03515519395de7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.468420 1029 tablet_service.cc:1468] Processing CreateTablet for tablet be73873da994402e84fff6aa2ded7525 (DEFAULT_TABLE table=default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.468793 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 09b46a12076e492e89d0c0178bcf0524 (DEFAULT_TABLE table=default.Table_60f4c59edd6e4f6aa0954a178bcaea7e [id=3b8967eb30ad46cdbf272136f23318f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.469167 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ffa14750f1b94c589779f7da0e764ccc (table default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685]). Expected version 1 got 0
I20250905 19:52:15.470295 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d1f668b9ae964b7ebf87726f734e3a40 with delete_type TABLET_DATA_DELETED (replaced by tablet be73873da994402e84fff6aa2ded7525) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.470723 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_cb2d5de6bdc74b069d93e22c8ee74bf9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.471294 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cb2d5de6bdc74b069d93e22c8ee74bf9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.472952 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_87bff647cebf4507bb0e6a6caf45f752" } new_table_name: "default.Table_0de80dfdff2546a2a39368585f13a394" modify_external_catalogs: true
I20250905 19:52:15.474439 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_332f3b495a6548caa4cc12a01588ee6d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.474943 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_332f3b495a6548caa4cc12a01588ee6d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.477948 753 tablet_service.cc:1468] Processing CreateTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (DEFAULT_TABLE table=default.Table_c9d143854de148aea480b85f2718ef23 [id=6b4e399e326f47ad8c3ca4d5e33538d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.478361 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09be30768f5742cf9a4bf1e3fe86dfe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.478709 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_36557fc6d91f4afb9816c01bb97c550c" } new_table_name: "default.Table_db38b4b80e674f46846361436677e883" modify_external_catalogs: true
I20250905 19:52:15.479362 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_5ce48afe3ba94518bd44699d763ac7a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.479878 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5ce48afe3ba94518bd44699d763ac7a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.480042 887 tablet_service.cc:1468] Processing CreateTablet for tablet be73873da994402e84fff6aa2ded7525 (DEFAULT_TABLE table=default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.481062 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_257ae7ff742645f8ba4c7ebc4cc7ccc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:15.481355 1171 tablet_bootstrap.cc:654] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:15.481562 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_257ae7ff742645f8ba4c7ebc4cc7ccc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.482578 1185 tablet_replica.cc:331] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.482781 1185 raft_consensus.cc:2241] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.482960 893 tablet_service.cc:1468] Processing CreateTablet for tablet d20e269bf2224fe49f701c4777c42632 (DEFAULT_TABLE table=default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.483387 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20e269bf2224fe49f701c4777c42632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.483371 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2ccbbb56da30450a9d81bed4eeadc8ef" } new_table_name: "default.Table_409df36c5b6f4d1aa96720b58745e361" modify_external_catalogs: true
I20250905 19:52:15.484896 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 9e421213b7bd4dd8a03515519395de7d (DEFAULT_TABLE table=default.Table_b4ad86cd8af24e86b952d94a60b5fa41 [id=152edbecfe3648659c48a2cf89ff9798]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.485083 748 tablet_service.cc:1468] Processing CreateTablet for tablet 1c6295c8c7874df89662b8cef6459fec (DEFAULT_TABLE table=default.Table_87bff647cebf4507bb0e6a6caf45f752 [id=2151f6f3348c4d1fb376448ecfcbae8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.485587 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6295c8c7874df89662b8cef6459fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.482983 1185 raft_consensus.cc:2270] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.488663 747 tablet_service.cc:1468] Processing CreateTablet for tablet be73873da994402e84fff6aa2ded7525 (DEFAULT_TABLE table=default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.489219 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be73873da994402e84fff6aa2ded7525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.489390 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:34395
--webserver_interface=127.0.104.60
--webserver_port=44671
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 19:52:15.491227 1185 ts_tablet_manager.cc:1905] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.492029 1024 tablet_service.cc:1468] Processing CreateTablet for tablet d20e269bf2224fe49f701c4777c42632 (DEFAULT_TABLE table=default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.492333 746 tablet_service.cc:1468] Processing CreateTablet for tablet 09b46a12076e492e89d0c0178bcf0524 (DEFAULT_TABLE table=default.Table_60f4c59edd6e4f6aa0954a178bcaea7e [id=3b8967eb30ad46cdbf272136f23318f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.492818 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b46a12076e492e89d0c0178bcf0524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.495095 1184 ts_tablet_manager.cc:1905] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.495164 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be73873da994402e84fff6aa2ded7525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.496354 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9e421213b7bd4dd8a03515519395de7d (DEFAULT_TABLE table=default.Table_b4ad86cd8af24e86b952d94a60b5fa41 [id=152edbecfe3648659c48a2cf89ff9798]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.496917 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e421213b7bd4dd8a03515519395de7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.499104 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be73873da994402e84fff6aa2ded7525. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.500663 1236 raft_consensus.cc:491] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.500798 1185 ts_tablet_manager.cc:1918] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.500917 1185 log.cc:1199] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/987a6193d2a74f90906a45b237040e39
I20250905 19:52:15.500830 1236 raft_consensus.cc:513] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.501322 1185 ts_tablet_manager.cc:1939] T 987a6193d2a74f90906a45b237040e39 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.501690 1236 leader_election.cc:290] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:15.501878 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c6295c8c7874df89662b8cef6459fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.503353 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 987a6193d2a74f90906a45b237040e39 (table default.Table_36557fc6d91f4afb9816c01bb97c550c [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:15.504776 1053 raft_consensus.cc:1273] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:15.504963 1052 raft_consensus.cc:1273] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:15.505391 1196 consensus_queue.cc:1035] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:15.508482 1185 tablet_replica.cc:331] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.508706 1185 raft_consensus.cc:2241] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.508906 1185 raft_consensus.cc:2270] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.510092 1196 consensus_queue.cc:1035] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:15.512840 1185 ts_tablet_manager.cc:1905] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.514796 914 raft_consensus.cc:1273] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:52:15.515303 440 catalog_manager.cc:5001] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): alter failed for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]),no further retry: Not found: Tablet not found: d1f668b9ae964b7ebf87726f734e3a40
W20250905 19:52:15.515653 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d1f668b9ae964b7ebf87726f734e3a40 because the tablet was not found. No further retry: Not found: Tablet not found: d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.516638 744 tablet_service.cc:1468] Processing CreateTablet for tablet d20e269bf2224fe49f701c4777c42632 (DEFAULT_TABLE table=default.Table_0b23b83842434d578f1044e4ff7b37f3 [id=5cdcc43b222346d481565b3d3655d540]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.517951 1196 consensus_queue.cc:1035] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:15.522779 1184 ts_tablet_manager.cc:1918] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.522943 1184 log.cc:1199] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/f842c6ac96ec44ffaae02edb4a9fdeef
I20250905 19:52:15.523283 1184 ts_tablet_manager.cc:1939] T f842c6ac96ec44ffaae02edb4a9fdeef P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:15.525940 1238 tablet.cc:1722] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:15.526454 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec810e5cc1af4b9d9f6bd421072bbc77" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:15.527611 1240 tablet.cc:1722] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:15.527725 1228 mvcc.cc:204] Tried to move back new op lower bound from 7197089527710715904 to 7197089526830469120. Current Snapshot: MvccSnapshot[applied={T|T < 7197089527710715904}]
W20250905 19:52:15.529052 986 leader_election.cc:343] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.529047 1223 mvcc.cc:204] Tried to move back new op lower bound from 7197089527710715904 to 7197089526830469120. Current Snapshot: MvccSnapshot[applied={T|T < 7197089527710715904}]
I20250905 19:52:15.533510 912 raft_consensus.cc:3058] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.536741 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet f842c6ac96ec44ffaae02edb4a9fdeef (table default.Table_36557fc6d91f4afb9816c01bb97c550c [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:15.539472 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20e269bf2224fe49f701c4777c42632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.539893 1185 ts_tablet_manager.cc:1918] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.540073 1185 log.cc:1199] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e25832a05f034c25b20b55adfdcb63d7
I20250905 19:52:15.540460 1185 ts_tablet_manager.cc:1939] T e25832a05f034c25b20b55adfdcb63d7 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.541162 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 09b46a12076e492e89d0c0178bcf0524. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.541502 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e25832a05f034c25b20b55adfdcb63d7 (table default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 [id=f483c42f9aea43f38894538f57e7db56]) successfully deleted
I20250905 19:52:15.544358 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e421213b7bd4dd8a03515519395de7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.550899 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d20e269bf2224fe49f701c4777c42632. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.551607 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec810e5cc1af4b9d9f6bd421072bbc77" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:15.552598 984 leader_election.cc:343] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.552786 984 leader_election.cc:304] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:15.553550 1236 raft_consensus.cc:2747] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:15.569092 1169 tablet_bootstrap.cc:492] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.570245 1169 ts_tablet_manager.cc:1397] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.167s user 0.004s sys 0.004s
I20250905 19:52:15.571431 1169 raft_consensus.cc:357] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.573386 1169 raft_consensus.cc:383] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.576073 1171 tablet_bootstrap.cc:492] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.576236 1171 ts_tablet_manager.cc:1397] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.136s user 0.012s sys 0.019s
I20250905 19:52:15.577080 1171 raft_consensus.cc:357] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.577286 1171 raft_consensus.cc:383] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.577375 1171 raft_consensus.cc:738] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.577630 1171 consensus_queue.cc:260] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.578302 1171 ts_tablet_manager.cc:1428] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:15.578583 1171 tablet_bootstrap.cc:492] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.580024 1240 tablet.cc:1722] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:15.580374 1169 raft_consensus.cc:738] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.580660 1169 consensus_queue.cc:260] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.581138 1169 ts_tablet_manager.cc:1428] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250905 19:52:15.582612 1171 tablet_bootstrap.cc:654] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.582744 1169 tablet_bootstrap.cc:492] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.584494 1169 tablet_bootstrap.cc:654] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.586714 1238 tablet.cc:1722] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:15.593634 1169 tablet_bootstrap.cc:492] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.593812 1169 ts_tablet_manager.cc:1397] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250905 19:52:15.594627 1171 tablet_bootstrap.cc:492] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.594710 1169 raft_consensus.cc:357] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.594890 1171 ts_tablet_manager.cc:1397] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250905 19:52:15.594890 1169 raft_consensus.cc:383] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.594980 1169 raft_consensus.cc:738] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.595274 1169 consensus_queue.cc:260] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.595638 1171 raft_consensus.cc:357] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.595860 1171 raft_consensus.cc:383] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.595947 1171 raft_consensus.cc:738] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.596242 1171 consensus_queue.cc:260] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.596808 1169 ts_tablet_manager.cc:1428] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:15.596834 1171 ts_tablet_manager.cc:1428] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:15.597134 1171 tablet_bootstrap.cc:492] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.597872 1169 tablet_bootstrap.cc:492] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.599443 1169 tablet_bootstrap.cc:654] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.601488 1171 tablet_bootstrap.cc:654] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:15.602978 440 catalog_manager.cc:5001] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): alter failed for tablet d1f668b9ae964b7ebf87726f734e3a40 (table default.Table_de29f307cc654a6880fba89258c177e7 [id=5190c11e4b1142909f87e75ce899c422]),no further retry: Not found: Tablet not found: d1f668b9ae964b7ebf87726f734e3a40
I20250905 19:52:15.603730 1236 raft_consensus.cc:491] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.603860 1236 raft_consensus.cc:513] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.604452 1236 leader_election.cc:290] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:15.605222 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf0b20630b44230b17a80cdb53322fa" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:15.605777 986 leader_election.cc:343] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.606283 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf0b20630b44230b17a80cdb53322fa" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.606405 912 raft_consensus.cc:1273] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20250905 19:52:15.606797 984 leader_election.cc:343] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.606912 984 leader_election.cc:304] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:15.607264 1236 raft_consensus.cc:2747] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:15.607602 1196 consensus_queue.cc:1035] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:15.615010 1171 tablet_bootstrap.cc:492] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.615175 1171 ts_tablet_manager.cc:1397] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.004s
I20250905 19:52:15.616101 1171 raft_consensus.cc:357] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.616288 1171 raft_consensus.cc:383] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.616380 1171 raft_consensus.cc:738] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.616637 1171 consensus_queue.cc:260] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.618077 1171 ts_tablet_manager.cc:1428] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:15.619382 1171 tablet_bootstrap.cc:492] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.619498 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 8adc74a357244102801b2aa8d27d9747
I20250905 19:52:15.620150 454 catalog_manager.cc:6652] ReplaceTablet: tablet 8adc74a357244102801b2aa8d27d9747 deleted and replaced by tablet 2201e4bad99643a9b041f6f99406433b
I20250905 19:52:15.621022 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 8adc74a357244102801b2aa8d27d9747 with delete_type TABLET_DATA_DELETED (replaced by tablet 2201e4bad99643a9b041f6f99406433b) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.621766 1171 tablet_bootstrap.cc:654] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.622180 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 8adc74a357244102801b2aa8d27d9747 with delete_type TABLET_DATA_DELETED (replaced by tablet 2201e4bad99643a9b041f6f99406433b) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:15.622697 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8adc74a357244102801b2aa8d27d9747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8adc74a357244102801b2aa8d27d9747 already in progress: creating tablet
W20250905 19:52:15.622964 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 8adc74a357244102801b2aa8d27d9747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8adc74a357244102801b2aa8d27d9747 already in progress: creating tablet
I20250905 19:52:15.626709 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 8adc74a357244102801b2aa8d27d9747 with delete_type TABLET_DATA_DELETED (replaced by tablet 2201e4bad99643a9b041f6f99406433b) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.627616 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0b23b83842434d578f1044e4ff7b37f3" } new_table_name: "default.Table_450fbb5afb874baba3ecfd350509f6f0" modify_external_catalogs: true
I20250905 19:52:15.629784 1180 raft_consensus.cc:491] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.629972 1180 raft_consensus.cc:513] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.630095 1239 tablet.cc:1722] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:15.630512 1180 leader_election.cc:290] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.631445 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afb404e5b06c452bb98abeea95070a5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:15.631649 1053 raft_consensus.cc:2466] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:15.632236 849 leader_election.cc:304] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:15.632731 1180 raft_consensus.cc:2802] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:15.632870 1180 raft_consensus.cc:491] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:15.633015 1180 raft_consensus.cc:3058] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.637267 1180 raft_consensus.cc:513] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.638340 1169 tablet_bootstrap.cc:492] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.638573 1169 ts_tablet_manager.cc:1397] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.041s user 0.000s sys 0.009s
I20250905 19:52:15.638612 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afb404e5b06c452bb98abeea95070a5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:15.638813 1053 raft_consensus.cc:3058] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.639638 1169 raft_consensus.cc:357] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.640014 1169 raft_consensus.cc:383] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.640120 1169 raft_consensus.cc:738] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.640501 1169 consensus_queue.cc:260] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.641988 1053 raft_consensus.cc:2466] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:15.642562 1169 ts_tablet_manager.cc:1428] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250905 19:52:15.642570 849 leader_election.cc:304] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:15.643316 1169 tablet_bootstrap.cc:492] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.643692 1272 raft_consensus.cc:2802] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:15.644170 1180 leader_election.cc:290] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.644536 1171 tablet_bootstrap.cc:492] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.644740 1171 ts_tablet_manager.cc:1397] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.004s
I20250905 19:52:15.645411 1169 tablet_bootstrap.cc:654] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.636618 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afb404e5b06c452bb98abeea95070a5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.645561 1171 raft_consensus.cc:357] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.643529 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "afb404e5b06c452bb98abeea95070a5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:15.645789 1171 raft_consensus.cc:383] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.645861 1171 raft_consensus.cc:738] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.646132 1171 consensus_queue.cc:260] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:15.646158 848 leader_election.cc:343] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:15.646628 848 leader_election.cc:343] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.647918 1171 ts_tablet_manager.cc:1428] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:15.648003 1272 raft_consensus.cc:695] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
W20250905 19:52:15.648089 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8adc74a357244102801b2aa8d27d9747 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8adc74a357244102801b2aa8d27d9747 already in progress: creating tablet
I20250905 19:52:15.648406 1171 tablet_bootstrap.cc:492] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.648785 1272 consensus_queue.cc:237] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.650892 1171 tablet_bootstrap.cc:654] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.653895 1169 tablet_bootstrap.cc:492] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.654116 1169 ts_tablet_manager.cc:1397] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250905 19:52:15.655149 1169 raft_consensus.cc:357] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.655405 1169 raft_consensus.cc:383] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.655608 1169 raft_consensus.cc:738] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.656008 1169 consensus_queue.cc:260] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.658238 1169 ts_tablet_manager.cc:1428] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:15.659739 1169 tablet_bootstrap.cc:492] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.661515 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8fe7ea747ff749c389bdd49a8291bf19" } modify_external_catalogs: true
I20250905 19:52:15.661710 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8fe7ea747ff749c389bdd49a8291bf19" } modify_external_catalogs: true
I20250905 19:52:15.662067 1169 tablet_bootstrap.cc:654] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.664498 1171 tablet_bootstrap.cc:492] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.664695 1171 ts_tablet_manager.cc:1397] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.007s
I20250905 19:52:15.665817 1171 raft_consensus.cc:357] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.666064 1171 raft_consensus.cc:383] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.666184 1171 raft_consensus.cc:738] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.666525 1171 consensus_queue.cc:260] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.667145 1171 ts_tablet_manager.cc:1428] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250905 19:52:15.667490 1171 tablet_bootstrap.cc:492] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.667769 1169 tablet_bootstrap.cc:492] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.667932 1169 ts_tablet_manager.cc:1397] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20250905 19:52:15.668898 1169 raft_consensus.cc:357] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.669112 1169 raft_consensus.cc:383] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.669209 1169 raft_consensus.cc:738] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.669271 1171 tablet_bootstrap.cc:654] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.669627 1169 consensus_queue.cc:260] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.671314 1169 ts_tablet_manager.cc:1428] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:15.672832 1169 tablet_bootstrap.cc:492] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.672992 448 master_service.cc:948] ReplaceTablet: received request to replace tablet d7d7567807594201a598d715ffbb318b from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.675313 1169 tablet_bootstrap.cc:654] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.681414 1171 tablet_bootstrap.cc:492] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.681617 1171 ts_tablet_manager.cc:1397] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20250905 19:52:15.682005 1169 tablet_bootstrap.cc:492] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.682158 1169 ts_tablet_manager.cc:1397] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.004s
I20250905 19:52:15.682507 1171 raft_consensus.cc:357] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.682796 1171 raft_consensus.cc:383] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.682880 1171 raft_consensus.cc:738] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.682942 1169 raft_consensus.cc:357] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.683115 1169 raft_consensus.cc:383] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.683205 1169 raft_consensus.cc:738] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.683146 1171 consensus_queue.cc:260] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.683450 1169 consensus_queue.cc:260] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.684386 1171 ts_tablet_manager.cc:1428] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:15.684701 1171 tablet_bootstrap.cc:492] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.684964 1169 ts_tablet_manager.cc:1428] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:15.686115 1169 tablet_bootstrap.cc:492] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.686534 1171 tablet_bootstrap.cc:654] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.687906 1169 tablet_bootstrap.cc:654] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.693413 1169 tablet_bootstrap.cc:492] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.693636 1169 ts_tablet_manager.cc:1397] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20250905 19:52:15.694687 1169 raft_consensus.cc:357] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.694948 1169 raft_consensus.cc:383] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.695070 1169 raft_consensus.cc:738] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.695426 1169 consensus_queue.cc:260] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.697486 1169 ts_tablet_manager.cc:1428] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250905 19:52:15.698990 1169 tablet_bootstrap.cc:492] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.701995 1169 tablet_bootstrap.cc:654] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.702147 1171 tablet_bootstrap.cc:492] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.702297 1171 ts_tablet_manager.cc:1397] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.008s
W20250905 19:52:15.702384 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1c6295c8c7874df89662b8cef6459fec consensus state has no leader
I20250905 19:52:15.703212 1171 raft_consensus.cc:357] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.703450 1171 raft_consensus.cc:383] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.703553 1171 raft_consensus.cc:738] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.703917 1171 consensus_queue.cc:260] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.706429 1171 ts_tablet_manager.cc:1428] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:15.706751 1171 tablet_bootstrap.cc:492] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.709015 1171 tablet_bootstrap.cc:654] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.710343 1272 raft_consensus.cc:491] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.710508 1272 raft_consensus.cc:513] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.710950 1169 tablet_bootstrap.cc:492] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.711134 1169 ts_tablet_manager.cc:1397] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.007s
I20250905 19:52:15.711160 1272 leader_election.cc:290] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.712162 1169 raft_consensus.cc:357] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.712386 1169 raft_consensus.cc:383] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.712486 1169 raft_consensus.cc:738] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.712819 1169 consensus_queue.cc:260] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.713893 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf0b20630b44230b17a80cdb53322fa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:15.714154 1053 raft_consensus.cc:2466] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:15.714632 849 leader_election.cc:304] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:15.714869 1169 ts_tablet_manager.cc:1428] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:15.715121 1272 raft_consensus.cc:2802] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:15.715217 1272 raft_consensus.cc:491] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:15.715312 1272 raft_consensus.cc:3058] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.716298 1169 tablet_bootstrap.cc:492] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.718730 1169 tablet_bootstrap.cc:654] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.719127 1272 raft_consensus.cc:513] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.721211 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf0b20630b44230b17a80cdb53322fa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.721375 1169 tablet_bootstrap.cc:492] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.721562 1169 ts_tablet_manager.cc:1397] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.005s user 0.002s sys 0.001s
I20250905 19:52:15.721635 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf0b20630b44230b17a80cdb53322fa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:15.722544 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbf0b20630b44230b17a80cdb53322fa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:15.722589 1169 raft_consensus.cc:357] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.722772 1053 raft_consensus.cc:3058] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.722864 1169 raft_consensus.cc:383] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.722955 1169 raft_consensus.cc:738] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.723331 1169 consensus_queue.cc:260] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.725390 1169 ts_tablet_manager.cc:1428] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250905 19:52:15.726840 1169 tablet_bootstrap.cc:492] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:15.727289 848 leader_election.cc:343] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:15.727766 848 leader_election.cc:343] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.728654 1053 raft_consensus.cc:2466] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:15.730273 1171 tablet_bootstrap.cc:492] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.730448 1171 ts_tablet_manager.cc:1397] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.008s sys 0.015s
I20250905 19:52:15.730790 849 leader_election.cc:304] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:15.731154 1273 raft_consensus.cc:2802] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:15.731316 1273 raft_consensus.cc:695] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:15.731539 1171 raft_consensus.cc:357] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.731747 1171 raft_consensus.cc:383] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.731835 1171 raft_consensus.cc:738] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.731662 1273 consensus_queue.cc:237] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.732154 1171 consensus_queue.cc:260] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.735280 1196 raft_consensus.cc:491] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.735530 1196 raft_consensus.cc:513] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.736496 1196 leader_election.cc:290] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:15.737636 712 leader_election.cc:343] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: e25832a05f034c25b20b55adfdcb63d7
W20250905 19:52:15.738085 711 leader_election.cc:343] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: e25832a05f034c25b20b55adfdcb63d7
I20250905 19:52:15.738227 711 leader_election.cc:304] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:15.738720 1196 raft_consensus.cc:2747] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:15.742257 1272 leader_election.cc:290] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:15.748154 1265 log.cc:927] Time spent T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.065s user 0.000s sys 0.000s
I20250905 19:52:15.749080 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
I20250905 19:52:15.729183 1169 tablet_bootstrap.cc:654] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.757885 1170 tablet_bootstrap.cc:492] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.758229 1170 ts_tablet_manager.cc:1397] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.351s user 0.002s sys 0.006s
I20250905 19:52:15.759095 1170 raft_consensus.cc:357] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.759361 1170 raft_consensus.cc:383] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.759486 1170 raft_consensus.cc:738] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.759786 1170 consensus_queue.cc:260] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.760360 1239 tablet.cc:1722] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:15.762913 1170 ts_tablet_manager.cc:1428] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:15.764853 1170 tablet_bootstrap.cc:492] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.765717 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_eb3258db08db42738115c037a4d2912d" } modify_external_catalogs: true
I20250905 19:52:15.765921 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_eb3258db08db42738115c037a4d2912d" } modify_external_catalogs: true
I20250905 19:52:15.768918 1170 tablet_bootstrap.cc:654] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.777652 1171 ts_tablet_manager.cc:1428] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.047s user 0.014s sys 0.031s
I20250905 19:52:15.777989 1171 tablet_bootstrap.cc:492] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.779770 1171 tablet_bootstrap.cc:654] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.782164 1171 tablet_bootstrap.cc:492] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.782272 1171 ts_tablet_manager.cc:1397] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20250905 19:52:15.782866 1171 raft_consensus.cc:357] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.783053 1171 raft_consensus.cc:383] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.783119 1171 raft_consensus.cc:738] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.783293 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 78ac5c7e9b6741d79db2e606d55a3509 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.783324 1171 consensus_queue.cc:260] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.783787 1171 ts_tablet_manager.cc:1428] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:15.784093 1171 tablet_bootstrap.cc:492] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.785732 1171 tablet_bootstrap.cc:654] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.788750 1170 tablet_bootstrap.cc:492] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.788914 1170 ts_tablet_manager.cc:1397] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.002s
I20250905 19:52:15.789692 1170 raft_consensus.cc:357] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.789918 1170 raft_consensus.cc:383] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.790035 1170 raft_consensus.cc:738] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.790300 1170 consensus_queue.cc:260] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.792768 1170 ts_tablet_manager.cc:1428] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:15.793274 1171 tablet_bootstrap.cc:492] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.793424 1171 ts_tablet_manager.cc:1397] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.001s
I20250905 19:52:15.793870 1170 tablet_bootstrap.cc:492] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.794158 1171 raft_consensus.cc:357] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.794338 1171 raft_consensus.cc:383] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.794404 1171 raft_consensus.cc:738] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.794657 1171 consensus_queue.cc:260] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.795284 1171 ts_tablet_manager.cc:1428] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.795667 1171 tablet_bootstrap.cc:492] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.797346 1170 tablet_bootstrap.cc:654] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.797596 1171 tablet_bootstrap.cc:654] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.799865 1169 tablet_bootstrap.cc:492] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.800143 1169 ts_tablet_manager.cc:1397] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.073s user 0.005s sys 0.001s
I20250905 19:52:15.801767 1169 raft_consensus.cc:357] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.802073 1169 raft_consensus.cc:383] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.802210 1169 raft_consensus.cc:738] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.802881 1169 consensus_queue.cc:260] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.804457 1169 ts_tablet_manager.cc:1428] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:15.804857 1169 tablet_bootstrap.cc:492] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.808320 1169 tablet_bootstrap.cc:654] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:15.810380 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59faa6716fed4ea6b563df5f024b2058 has no consensus state
I20250905 19:52:15.810873 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ce8e455023774d02ba896de13623c0ea" } modify_external_catalogs: true
I20250905 19:52:15.811141 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ce8e455023774d02ba896de13623c0ea" } modify_external_catalogs: true
I20250905 19:52:15.817024 1171 tablet_bootstrap.cc:492] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:15.817183 1171 ts_tablet_manager.cc:1397] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.010s sys 0.010s
I20250905 19:52:15.817962 1171 raft_consensus.cc:357] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.818152 1171 raft_consensus.cc:383] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.818249 1171 raft_consensus.cc:738] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.818538 1171 consensus_queue.cc:260] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.819185 1171 ts_tablet_manager.cc:1428] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.820196 1171 tablet_bootstrap.cc:492] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:15.821178 453 catalog_manager.cc:5582] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:15.821592 453 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:15.821900 1171 tablet_bootstrap.cc:654] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.824712 449 catalog_manager.cc:5582] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:15.825103 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet cbf0b20630b44230b17a80cdb53322fa (table default.Table_eb3258db08db42738115c037a4d2912d [id=3ce5d572e4104f8cb8100e115eaa8df0]). Expected version 1 got 0
I20250905 19:52:15.825362 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_edca9fb5f8fb48d28540ecd7e385178c [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 3 got 0
I20250905 19:52:15.825789 1170 tablet_bootstrap.cc:492] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.826073 1170 ts_tablet_manager.cc:1397] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.001s
I20250905 19:52:15.826881 1170 raft_consensus.cc:357] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.827103 1170 raft_consensus.cc:383] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.827174 1170 raft_consensus.cc:738] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:15.827643 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f8d3cbdda0c94e139259d90919dc2f95 consensus state has no leader
I20250905 19:52:15.827548 1170 consensus_queue.cc:260] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.829680 1169 tablet_bootstrap.cc:492] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.829896 1169 ts_tablet_manager.cc:1397] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.025s user 0.002s sys 0.006s
I20250905 19:52:15.830183 1170 ts_tablet_manager.cc:1428] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:15.830752 1169 raft_consensus.cc:357] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.830943 1169 raft_consensus.cc:383] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.831020 1169 raft_consensus.cc:738] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.831346 1170 tablet_bootstrap.cc:492] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.831288 1169 consensus_queue.cc:260] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.831799 1169 ts_tablet_manager.cc:1428] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:15.832100 1169 tablet_bootstrap.cc:492] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:15.833230 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef with delete_type TABLET_DATA_DELETED (replaced by tablet f8d3cbdda0c94e139259d90919dc2f95) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.833539 744 tablet_service.cc:1515] Processing DeleteTablet for tablet e25832a05f034c25b20b55adfdcb63d7 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.833837 1186 tablet_replica.cc:331] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:15.834082 1186 raft_consensus.cc:2241] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.834295 1186 raft_consensus.cc:2270] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.835006 1186 ts_tablet_manager.cc:1905] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:15.838625 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d20e269bf2224fe49f701c4777c42632 consensus state has no leader
I20250905 19:52:15.840404 890 tablet_service.cc:1468] Processing CreateTablet for tablet 2c884954e5dc4445a12252953593d0bc (DEFAULT_TABLE table=default.Table_164521ea6f9342c8b5380153cb22b888 [id=26cde3c02a704bb4931f65f8dba00173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.840914 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c884954e5dc4445a12252953593d0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.844185 1170 tablet_bootstrap.cc:654] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.845381 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 66df0522aa2842f68aa824571fa61b05 (DEFAULT_TABLE table=default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3 [id=53917d92e1214845a54503f40686b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.845780 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66df0522aa2842f68aa824571fa61b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.845754 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2c884954e5dc4445a12252953593d0bc (DEFAULT_TABLE table=default.Table_164521ea6f9342c8b5380153cb22b888 [id=26cde3c02a704bb4931f65f8dba00173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.846174 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 59faa6716fed4ea6b563df5f024b2058 (DEFAULT_TABLE table=default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.846585 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 1664cbe248514535953a1ef2d312f98f (DEFAULT_TABLE table=default.Table_6a14b2be50f942d8bfd8c47247512879 [id=353aa71a6ee64c41aa94daadd629d845]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.847012 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cf2881d228964cd3bef040a06045d8d2 (DEFAULT_TABLE table=default.Table_9554a085b43c42a98edcb684be28faaa [id=1656bb34c67340f4b3793d8ffdf3a16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.847414 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 (DEFAULT_TABLE table=default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.849205 452 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 2201e4bad99643a9b041f6f99406433b
I20250905 19:52:15.849602 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6c729e17dd76402eb3cd6ed85f3d9875" } modify_external_catalogs: true
I20250905 19:52:15.849864 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6c729e17dd76402eb3cd6ed85f3d9875" } modify_external_catalogs: true
I20250905 19:52:15.851878 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 (DEFAULT_TABLE table=default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.852269 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6a14b2be50f942d8bfd8c47247512879" } modify_external_catalogs: true
I20250905 19:52:15.852533 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6a14b2be50f942d8bfd8c47247512879" } modify_external_catalogs: true
I20250905 19:52:15.853996 893 tablet_service.cc:1468] Processing CreateTablet for tablet 59faa6716fed4ea6b563df5f024b2058 (DEFAULT_TABLE table=default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.854427 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59faa6716fed4ea6b563df5f024b2058. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.856324 892 tablet_service.cc:1468] Processing CreateTablet for tablet 1664cbe248514535953a1ef2d312f98f (DEFAULT_TABLE table=default.Table_6a14b2be50f942d8bfd8c47247512879 [id=353aa71a6ee64c41aa94daadd629d845]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.856848 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1664cbe248514535953a1ef2d312f98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.860325 887 tablet_service.cc:1468] Processing CreateTablet for tablet 66df0522aa2842f68aa824571fa61b05 (DEFAULT_TABLE table=default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3 [id=53917d92e1214845a54503f40686b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.860702 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66df0522aa2842f68aa824571fa61b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.863129 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_60f4c59edd6e4f6aa0954a178bcaea7e" } new_table_name: "default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a" modify_external_catalogs: true
I20250905 19:52:15.864665 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c884954e5dc4445a12252953593d0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.869431 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59faa6716fed4ea6b563df5f024b2058. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.872342 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1664cbe248514535953a1ef2d312f98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.874202 1186 ts_tablet_manager.cc:1918] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.877143 891 tablet_service.cc:1468] Processing CreateTablet for tablet cf2881d228964cd3bef040a06045d8d2 (DEFAULT_TABLE table=default.Table_9554a085b43c42a98edcb684be28faaa [id=1656bb34c67340f4b3793d8ffdf3a16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.877197 1285 raft_consensus.cc:491] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.877393 1285 raft_consensus.cc:513] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.877625 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2881d228964cd3bef040a06045d8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.878334 1285 leader_election.cc:290] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.880244 894 tablet_service.cc:1468] Processing CreateTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 (DEFAULT_TABLE table=default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.880625 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57259d8eb0d44d1cbac80047c7ad67b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.879120 748 tablet_service.cc:1468] Processing CreateTablet for tablet 59faa6716fed4ea6b563df5f024b2058 (DEFAULT_TABLE table=default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.882615 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59faa6716fed4ea6b563df5f024b2058. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.884436 888 tablet_service.cc:1468] Processing CreateTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 (DEFAULT_TABLE table=default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.884866 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d3cbdda0c94e139259d90919dc2f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.879233 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5a7c8a1fe604706be88d0461b241a31" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.887305 1285 raft_consensus.cc:491] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:15.887398 776 raft_consensus.cc:2466] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:15.887465 1285 raft_consensus.cc:513] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.879508 747 tablet_service.cc:1468] Processing CreateTablet for tablet 1664cbe248514535953a1ef2d312f98f (DEFAULT_TABLE table=default.Table_6a14b2be50f942d8bfd8c47247512879 [id=353aa71a6ee64c41aa94daadd629d845]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.888067 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1664cbe248514535953a1ef2d312f98f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.888129 1285 leader_election.cc:290] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.888481 848 leader_election.cc:304] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:15.889254 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e62dbe9e48544109446323b2cc4501f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:15.889786 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5a7c8a1fe604706be88d0461b241a31" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:15.889873 848 leader_election.cc:343] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:15.889997 1053 raft_consensus.cc:2466] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:15.890151 1285 raft_consensus.cc:2802] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:15.890239 1285 raft_consensus.cc:491] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:15.890350 1285 raft_consensus.cc:3058] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.879848 746 tablet_service.cc:1468] Processing CreateTablet for tablet cf2881d228964cd3bef040a06045d8d2 (DEFAULT_TABLE table=default.Table_9554a085b43c42a98edcb684be28faaa [id=1656bb34c67340f4b3793d8ffdf3a16c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.892290 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2881d228964cd3bef040a06045d8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.893133 1285 raft_consensus.cc:513] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.894551 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5a7c8a1fe604706be88d0461b241a31" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:15.894723 776 raft_consensus.cc:3058] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.897569 776 raft_consensus.cc:2466] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:15.898079 848 leader_election.cc:304] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:15.880218 745 tablet_service.cc:1468] Processing CreateTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 (DEFAULT_TABLE table=default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.898386 1273 raft_consensus.cc:2802] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:15.880581 750 tablet_service.cc:1468] Processing CreateTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 (DEFAULT_TABLE table=default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.898540 1273 raft_consensus.cc:695] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:15.881098 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2c884954e5dc4445a12252953593d0bc (DEFAULT_TABLE table=default.Table_164521ea6f9342c8b5380153cb22b888 [id=26cde3c02a704bb4931f65f8dba00173]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.898636 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d3cbdda0c94e139259d90919dc2f95. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.898787 1273 consensus_queue.cc:237] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.898931 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c884954e5dc4445a12252953593d0bc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.901742 1285 leader_election.cc:290] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:15.904264 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57259d8eb0d44d1cbac80047c7ad67b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.881424 755 tablet_service.cc:1468] Processing CreateTablet for tablet 66df0522aa2842f68aa824571fa61b05 (DEFAULT_TABLE table=default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3 [id=53917d92e1214845a54503f40686b0e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:15.905208 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 66df0522aa2842f68aa824571fa61b05. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.935945 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cf2881d228964cd3bef040a06045d8d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.936389 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8d3cbdda0c94e139259d90919dc2f95. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:15.937327 849 leader_election.cc:343] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 7e62dbe9e48544109446323b2cc4501f
I20250905 19:52:15.937444 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5a7c8a1fe604706be88d0461b241a31" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:15.937541 849 leader_election.cc:304] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:15.937642 1053 raft_consensus.cc:3058] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:15.938082 1273 raft_consensus.cc:2747] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:15.939085 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57259d8eb0d44d1cbac80047c7ad67b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:15.943217 453 master_service.cc:948] ReplaceTablet: received request to replace tablet d7d7567807594201a598d715ffbb318b from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:15.943645 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 78ac5c7e9b6741d79db2e606d55a3509
I20250905 19:52:15.944262 451 catalog_manager.cc:6652] ReplaceTablet: tablet 78ac5c7e9b6741d79db2e606d55a3509 deleted and replaced by tablet 59c13a5b598a4597931ddf5d476aec85
I20250905 19:52:15.949275 1053 raft_consensus.cc:2466] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:15.953213 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c13a5b598a4597931ddf5d476aec85) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:15.954752 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 78ac5c7e9b6741d79db2e606d55a3509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78ac5c7e9b6741d79db2e606d55a3509 already in progress: creating tablet
I20250905 19:52:15.959885 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c13a5b598a4597931ddf5d476aec85) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:15.960831 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 78ac5c7e9b6741d79db2e606d55a3509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78ac5c7e9b6741d79db2e606d55a3509 already in progress: creating tablet
I20250905 19:52:15.966173 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet b776063b81c44217b517a619af927c08
I20250905 19:52:15.970096 454 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_edca9fb5f8fb48d28540ecd7e385178c [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 3 got 0
I20250905 19:52:15.971580 892 tablet_service.cc:1515] Processing DeleteTablet for tablet b776063b81c44217b517a619af927c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.972040 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet b776063b81c44217b517a619af927c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:15.972582 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet b776063b81c44217b517a619af927c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b776063b81c44217b517a619af927c08 already in progress: creating tablet
W20250905 19:52:15.972967 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet b776063b81c44217b517a619af927c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b776063b81c44217b517a619af927c08 already in progress: creating tablet
I20250905 19:52:15.973925 1169 tablet_bootstrap.cc:654] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:15.974309 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d7d7567807594201a598d715ffbb318b
I20250905 19:52:15.974960 448 catalog_manager.cc:6652] ReplaceTablet: tablet d7d7567807594201a598d715ffbb318b deleted and replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:15.976109 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d7567807594201a598d715ffbb318b with delete_type TABLET_DATA_DELETED (replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:15.976675 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet d7d7567807594201a598d715ffbb318b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7d7567807594201a598d715ffbb318b already in progress: creating tablet
I20250905 19:52:15.976764 887 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d7567807594201a598d715ffbb318b with delete_type TABLET_DATA_DELETED (replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.977078 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c13a5b598a4597931ddf5d476aec85) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.977270 755 tablet_service.cc:1515] Processing DeleteTablet for tablet b776063b81c44217b517a619af927c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.977375 750 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d7567807594201a598d715ffbb318b with delete_type TABLET_DATA_DELETED (replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.977461 1186 log.cc:1199] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/f842c6ac96ec44ffaae02edb4a9fdeef
W20250905 19:52:15.977818 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet d7d7567807594201a598d715ffbb318b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7d7567807594201a598d715ffbb318b already in progress: creating tablet
I20250905 19:52:15.977885 1186 ts_tablet_manager.cc:1939] T f842c6ac96ec44ffaae02edb4a9fdeef P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:15.979012 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet f842c6ac96ec44ffaae02edb4a9fdeef (table default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:15.979372 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 0480eda7e20c4336bbeacad1e7cdd56a
I20250905 19:52:15.980616 1186 tablet_replica.cc:331] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:15.982525 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:15.982784 1184 tablet_replica.cc:331] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:15.982964 1184 raft_consensus.cc:2241] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.983142 1184 raft_consensus.cc:2270] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.980970 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:15.983392 1186 raft_consensus.cc:2241] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.983443 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:15.983579 1186 raft_consensus.cc:2270] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.983835 1185 tablet_replica.cc:331] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:15.983916 1184 ts_tablet_manager.cc:1905] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.984062 1185 raft_consensus.cc:2241] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:15.984237 1185 raft_consensus.cc:2270] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:15.984225 1186 ts_tablet_manager.cc:1905] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.984894 1185 ts_tablet_manager.cc:1905] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:15.990427 1184 ts_tablet_manager.cc:1918] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.990571 1184 log.cc:1199] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/0480eda7e20c4336bbeacad1e7cdd56a
I20250905 19:52:15.990958 1184 ts_tablet_manager.cc:1939] T 0480eda7e20c4336bbeacad1e7cdd56a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:15.992034 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 0480eda7e20c4336bbeacad1e7cdd56a (table default.Table_6c729e17dd76402eb3cd6ed85f3d9875 [id=d9c9bed59ab9499393be43c41657e6e0]) successfully deleted
I20250905 19:52:15.992779 1185 ts_tablet_manager.cc:1918] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:15.993304 1185 log.cc:1199] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/0480eda7e20c4336bbeacad1e7cdd56a
I20250905 19:52:15.993428 1170 tablet_bootstrap.cc:492] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:15.993599 1170 ts_tablet_manager.cc:1397] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.162s user 0.008s sys 0.021s
I20250905 19:52:15.993813 1169 tablet_bootstrap.cc:492] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:15.993959 1169 ts_tablet_manager.cc:1397] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.162s user 0.003s sys 0.000s
I20250905 19:52:15.994215 1185 ts_tablet_manager.cc:1939] T 0480eda7e20c4336bbeacad1e7cdd56a P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:15.994550 1170 raft_consensus.cc:357] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.994761 1170 raft_consensus.cc:383] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.994867 1170 raft_consensus.cc:738] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.994817 1169 raft_consensus.cc:357] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.994982 1169 raft_consensus.cc:383] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:15.995065 1169 raft_consensus.cc:738] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:15.995150 1170 consensus_queue.cc:260] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.995304 1169 consensus_queue.cc:260] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:15.996019 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_39151a7d21ac4d49bb5728f518e3db4f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:15.996693 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_39151a7d21ac4d49bb5728f518e3db4f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:15.997337 1170 ts_tablet_manager.cc:1428] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:15.997822 1170 tablet_bootstrap.cc:492] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:15.998445 1169 ts_tablet_manager.cc:1428] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250905 19:52:15.999377 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_feeb076a5fc84dd38f2fa2908e15b6b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:15.999547 1169 tablet_bootstrap.cc:492] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.000316 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1fd86ae5263542ed92214968ec16ccf0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.000524 1170 tablet_bootstrap.cc:654] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.000939 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1fd86ae5263542ed92214968ec16ccf0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.001277 1169 tablet_bootstrap.cc:654] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.003134 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f3bcb1e148584db3b4336af7b8028437"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.003746 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f3bcb1e148584db3b4336af7b8028437 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:16.007925 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_feeb076a5fc84dd38f2fa2908e15b6b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.010835 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 0480eda7e20c4336bbeacad1e7cdd56a (table default.Table_6c729e17dd76402eb3cd6ed85f3d9875 [id=d9c9bed59ab9499393be43c41657e6e0]) successfully deleted
I20250905 19:52:16.015763 1186 ts_tablet_manager.cc:1918] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.015923 1186 log.cc:1199] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e25832a05f034c25b20b55adfdcb63d7
I20250905 19:52:16.016412 1186 ts_tablet_manager.cc:1939] T e25832a05f034c25b20b55adfdcb63d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.017422 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet e25832a05f034c25b20b55adfdcb63d7 (table default.Table_3db28a103a6f46f08d3aeaf8fb3432d5 [id=f483c42f9aea43f38894538f57e7db56]) successfully deleted
I20250905 19:52:16.019680 1169 tablet_bootstrap.cc:492] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.019878 1169 ts_tablet_manager.cc:1397] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250905 19:52:16.020179 1170 tablet_bootstrap.cc:492] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:16.020339 1170 ts_tablet_manager.cc:1397] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.000s
I20250905 19:52:16.021129 1170 raft_consensus.cc:357] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.021339 1170 raft_consensus.cc:383] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.021416 1170 raft_consensus.cc:738] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.021620 1169 raft_consensus.cc:357] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.021689 1170 consensus_queue.cc:260] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.021852 1169 raft_consensus.cc:383] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.021929 1169 raft_consensus.cc:738] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.022197 1169 consensus_queue.cc:260] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.022562 1170 ts_tablet_manager.cc:1428] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:16.022917 1170 tablet_bootstrap.cc:492] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:16.023746 1169 ts_tablet_manager.cc:1428] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:16.024998 1171 tablet_bootstrap.cc:492] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.025174 1171 ts_tablet_manager.cc:1397] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.205s user 0.047s sys 0.071s
I20250905 19:52:16.025947 1171 raft_consensus.cc:357] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.026155 1171 raft_consensus.cc:383] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.026227 1171 raft_consensus.cc:738] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.026463 1171 consensus_queue.cc:260] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.026739 1175 raft_consensus.cc:491] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.026912 1175 raft_consensus.cc:513] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.027451 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 1664cbe248514535953a1ef2d312f98f
I20250905 19:52:16.027529 1175 leader_election.cc:290] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.028307 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8240e51124a74b81a0bdabb3ef35e66f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:16.028249 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ff5415e10957422498084d7730365989"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.028527 1053 raft_consensus.cc:2466] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
W20250905 19:52:16.028796 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ff5415e10957422498084d7730365989 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.028980 711 leader_election.cc:304] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:16.029275 1175 raft_consensus.cc:2802] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.029371 1175 raft_consensus.cc:491] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.029486 1175 raft_consensus.cc:3058] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:16.030079 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09b46a12076e492e89d0c0178bcf0524 consensus state has no leader
I20250905 19:52:16.030334 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 6f4ec474911941b2a2d0825cdb37bd72 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.030766 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 7e62dbe9e48544109446323b2cc4501f from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.031260 1171 ts_tablet_manager.cc:1428] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:16.031277 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b86b0c6f5fe4496ead402827d2db3f1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.031502 1170 tablet_bootstrap.cc:654] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.031752 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b86b0c6f5fe4496ead402827d2db3f1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.033432 1175 raft_consensus.cc:513] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.034158 1171 tablet_bootstrap.cc:492] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.034384 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8240e51124a74b81a0bdabb3ef35e66f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.034596 914 raft_consensus.cc:2466] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:16.034801 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8240e51124a74b81a0bdabb3ef35e66f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:16.034996 1053 raft_consensus.cc:3058] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.036465 1171 tablet_bootstrap.cc:654] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.036556 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 1664cbe248514535953a1ef2d312f98f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.037671 1053 raft_consensus.cc:2466] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:16.034394 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8240e51124a74b81a0bdabb3ef35e66f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:16.039767 913 raft_consensus.cc:3058] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.040006 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 1664cbe248514535953a1ef2d312f98f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.040792 711 leader_election.cc:304] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:16.041052 1203 raft_consensus.cc:2802] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.041184 1203 raft_consensus.cc:695] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:16.041429 1203 consensus_queue.cc:237] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:16.042487 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 1664cbe248514535953a1ef2d312f98f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1664cbe248514535953a1ef2d312f98f already in progress: creating tablet
I20250905 19:52:16.046465 1175 leader_election.cc:290] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.046528 913 raft_consensus.cc:2466] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:16.048125 1169 tablet_bootstrap.cc:492] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.049072 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 1664cbe248514535953a1ef2d312f98f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.049897 1169 tablet_bootstrap.cc:654] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.051347 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 78ac5c7e9b6741d79db2e606d55a3509 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 78ac5c7e9b6741d79db2e606d55a3509 already in progress: creating tablet
W20250905 19:52:16.051671 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet b776063b81c44217b517a619af927c08 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b776063b81c44217b517a619af927c08 already in progress: creating tablet
W20250905 19:52:16.051905 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d7d7567807594201a598d715ffbb318b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d7d7567807594201a598d715ffbb318b already in progress: creating tablet
I20250905 19:52:16.052712 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_fe03bfebc206445e842984b466de6d91"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.053501 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe03bfebc206445e842984b466de6d91 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:16.055008 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0480eda7e20c4336bbeacad1e7cdd56a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0480eda7e20c4336bbeacad1e7cdd56a already in progress: creating tablet
W20250905 19:52:16.055322 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 1664cbe248514535953a1ef2d312f98f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1664cbe248514535953a1ef2d312f98f already in progress: creating tablet
W20250905 19:52:16.056098 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 1664cbe248514535953a1ef2d312f98f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1664cbe248514535953a1ef2d312f98f already in progress: creating tablet
I20250905 19:52:16.063079 1170 tablet_bootstrap.cc:492] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:16.063309 1170 ts_tablet_manager.cc:1397] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250905 19:52:16.064190 1170 raft_consensus.cc:357] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.064875 1169 tablet_bootstrap.cc:492] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.065043 1169 ts_tablet_manager.cc:1397] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250905 19:52:16.066200 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_257ae7ff742645f8ba4c7ebc4cc7ccc2" } new_table_name: "default.Table_e24a01de544e4e12a3e017bdb595b067" modify_external_catalogs: true
I20250905 19:52:16.066426 1169 raft_consensus.cc:357] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.066619 1169 raft_consensus.cc:383] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.066509 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_528007b5c2c544e7a81752f8b473d78c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.066699 1169 raft_consensus.cc:738] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.066924 1169 consensus_queue.cc:260] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:16.067265 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_528007b5c2c544e7a81752f8b473d78c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.067443 1169 ts_tablet_manager.cc:1428] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:16.067847 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e73ba1d781524aa7a6c1714e3af86d7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.068326 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e73ba1d781524aa7a6c1714e3af86d7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.068629 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_fd280a2a9f5f4f81ad7dcd3fd2b4003b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.069231 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fd280a2a9f5f4f81ad7dcd3fd2b4003b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.071889 1170 raft_consensus.cc:383] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.072099 1170 raft_consensus.cc:738] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.072415 1170 consensus_queue.cc:260] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.074191 1176 raft_consensus.cc:491] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.074380 1176 raft_consensus.cc:513] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.075048 1176 leader_election.cc:290] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.076500 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2c7906a718c46d9a76587a70d96a1f9" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.076550 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2c7906a718c46d9a76587a70d96a1f9" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.076779 913 raft_consensus.cc:2466] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:16.077253 986 leader_election.cc:304] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.077538 1176 raft_consensus.cc:2802] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.077670 1176 raft_consensus.cc:491] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.077792 1176 raft_consensus.cc:3058] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:16.078384 984 leader_election.cc:343] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.080407 1169 tablet_bootstrap.cc:492] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.081792 1176 raft_consensus.cc:513] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.082346 1169 tablet_bootstrap.cc:654] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.083153 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2c7906a718c46d9a76587a70d96a1f9" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:16.083441 913 raft_consensus.cc:3058] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.086061 1170 ts_tablet_manager.cc:1428] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250905 19:52:16.087242 913 raft_consensus.cc:2466] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:16.087411 1170 tablet_bootstrap.cc:492] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:16.088322 986 leader_election.cc:304] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.088738 1236 raft_consensus.cc:2802] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.091126 1170 tablet_bootstrap.cc:654] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.092602 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2c7906a718c46d9a76587a70d96a1f9" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:16.093678 984 leader_election.cc:343] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.094374 1176 leader_election.cc:290] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.099385 1170 tablet_bootstrap.cc:492] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:16.099553 1170 ts_tablet_manager.cc:1397] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250905 19:52:16.100329 1170 raft_consensus.cc:357] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.100525 1170 raft_consensus.cc:383] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.100606 1170 raft_consensus.cc:738] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.100973 1170 consensus_queue.cc:260] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.106252 1236 raft_consensus.cc:695] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:16.106791 1236 consensus_queue.cc:237] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.107795 1170 ts_tablet_manager.cc:1428] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:16.111657 1169 tablet_bootstrap.cc:492] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.111837 1169 ts_tablet_manager.cc:1397] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.002s
I20250905 19:52:16.112846 1169 raft_consensus.cc:357] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.113044 1169 raft_consensus.cc:383] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.113137 1169 raft_consensus.cc:738] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.113400 1169 consensus_queue.cc:260] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.113907 1169 ts_tablet_manager.cc:1428] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:16.114562 1169 tablet_bootstrap.cc:492] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.116207 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_eb7fe5cbd839447e84d08c2f04f4b051" } new_table_name: "default.Table_6eb260fb20e14dc59c7f4bdb683fa538" modify_external_catalogs: true
I20250905 19:52:16.116612 1169 tablet_bootstrap.cc:654] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.120610 1171 tablet_bootstrap.cc:492] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.120918 1171 ts_tablet_manager.cc:1397] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.087s user 0.015s sys 0.039s
I20250905 19:52:16.122066 1171 raft_consensus.cc:357] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.123548 1171 raft_consensus.cc:383] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.123756 1171 raft_consensus.cc:738] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.125262 1171 consensus_queue.cc:260] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.127153 1171 ts_tablet_manager.cc:1428] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:16.128468 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_de29f307cc654a6880fba89258c177e7" } new_table_name: "default.Table_6999a7918e524a0293956f96df1caa80" modify_external_catalogs: true
I20250905 19:52:16.128727 1171 tablet_bootstrap.cc:492] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.131098 1171 tablet_bootstrap.cc:654] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.132370 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet cbf0b20630b44230b17a80cdb53322fa
I20250905 19:52:16.132989 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e5751a29750a47e8b4b75af6d76edd5d" } new_table_name: "default.Table_e88b4d40afe0429daf95a8650aee6126" modify_external_catalogs: true
I20250905 19:52:16.133231 1169 tablet_bootstrap.cc:492] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.133350 1169 ts_tablet_manager.cc:1397] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.016s
I20250905 19:52:16.133978 1169 raft_consensus.cc:357] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.134102 1169 raft_consensus.cc:383] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.134107 449 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685] alter complete (version 1)
I20250905 19:52:16.134166 1169 raft_consensus.cc:738] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.134363 1169 consensus_queue.cc:260] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.134848 1169 ts_tablet_manager.cc:1428] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:16.135145 1169 tablet_bootstrap.cc:492] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.135779 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_363859b13b284790a71ced21040081e4" } modify_external_catalogs: true
I20250905 19:52:16.136488 454 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e] alter complete (version 1)
I20250905 19:52:16.137451 1169 tablet_bootstrap.cc:654] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.138130 887 tablet_service.cc:1468] Processing CreateTablet for tablet 607c8a8848304491bbd4a7971f96322f (DEFAULT_TABLE table=default.Table_cb2d5de6bdc74b069d93e22c8ee74bf9 [id=f0398b74058b4ccbac71a2cb2f584a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.138353 891 tablet_service.cc:1468] Processing CreateTablet for tablet a884eca0e54a44dc920114e61a199689 (DEFAULT_TABLE table=default.Table_5ce48afe3ba94518bd44699d763ac7a9 [id=8c67898a816046e084be67f08f5edb71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.138610 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607c8a8848304491bbd4a7971f96322f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.138832 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a884eca0e54a44dc920114e61a199689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.138839 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f (DEFAULT_TABLE table=default.Table_363859b13b284790a71ced21040081e4 [id=3d169e77e70f4a059cdf41f2f3a70171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.139190 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6daaa0fe43414299c032d1f4d9bb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.139171 892 tablet_service.cc:1468] Processing CreateTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f (DEFAULT_TABLE table=default.Table_257ae7ff742645f8ba4c7ebc4cc7ccc2 [id=4044052e369843d39980a31b1654628d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.139577 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ae86cb11e24d348e2f12e982f3ab9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.139504 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f (DEFAULT_TABLE table=default.Table_363859b13b284790a71ced21040081e4 [id=3d169e77e70f4a059cdf41f2f3a70171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.139894 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6daaa0fe43414299c032d1f4d9bb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.144342 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a884eca0e54a44dc920114e61a199689 (DEFAULT_TABLE table=default.Table_5ce48afe3ba94518bd44699d763ac7a9 [id=8c67898a816046e084be67f08f5edb71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.144495 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f (DEFAULT_TABLE table=default.Table_257ae7ff742645f8ba4c7ebc4cc7ccc2 [id=4044052e369843d39980a31b1654628d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.144760 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a884eca0e54a44dc920114e61a199689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.146378 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ae86cb11e24d348e2f12e982f3ab9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.148294 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 607c8a8848304491bbd4a7971f96322f (DEFAULT_TABLE table=default.Table_cb2d5de6bdc74b069d93e22c8ee74bf9 [id=f0398b74058b4ccbac71a2cb2f584a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.148669 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607c8a8848304491bbd4a7971f96322f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.146395 1236 raft_consensus.cc:491] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.152726 1236 raft_consensus.cc:513] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.153362 1236 leader_election.cc:290] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.153908 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f1d960461ef4829960baa5cca7f7765" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:16.154541 984 leader_election.cc:343] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.154691 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf0b20630b44230b17a80cdb53322fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.155026 1184 tablet_replica.cc:331] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.155246 1184 raft_consensus.cc:2241] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.155411 1184 raft_consensus.cc:2270] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.155644 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.156029 1184 ts_tablet_manager.cc:1905] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.156095 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.156621 1185 tablet_replica.cc:331] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.156826 1185 raft_consensus.cc:2241] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.156996 1185 raft_consensus.cc:2270] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.155900 894 tablet_service.cc:1515] Processing DeleteTablet for tablet afb404e5b06c452bb98abeea95070a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.157523 888 tablet_service.cc:1515] Processing DeleteTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 with delete_type TABLET_DATA_DELETED (replaced by tablet 8adc74a357244102801b2aa8d27d9747) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.158090 1185 ts_tablet_manager.cc:1905] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.158066 1175 raft_consensus.cc:491] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.158236 1175 raft_consensus.cc:513] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.158895 1175 leader_election.cc:290] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.159508 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f1d960461ef4829960baa5cca7f7765" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.159719 913 raft_consensus.cc:2466] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:16.160890 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec810e5cc1af4b9d9f6bd421072bbc77" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.161626 914 raft_consensus.cc:2466] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:16.162096 986 leader_election.cc:304] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.162223 712 leader_election.cc:304] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:16.162817 1236 raft_consensus.cc:2802] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.162907 1236 raft_consensus.cc:491] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.162992 1236 raft_consensus.cc:3058] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.163223 1175 raft_consensus.cc:2802] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.163316 1175 raft_consensus.cc:491] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.163412 1175 raft_consensus.cc:3058] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.163739 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec810e5cc1af4b9d9f6bd421072bbc77" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:16.163928 1053 raft_consensus.cc:2466] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:16.155541 891 tablet_service.cc:1515] Processing DeleteTablet for tablet f842c6ac96ec44ffaae02edb4a9fdeef with delete_type TABLET_DATA_DELETED (replaced by tablet f8d3cbdda0c94e139259d90919dc2f95) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.165062 1185 ts_tablet_manager.cc:1918] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.165211 1185 log.cc:1199] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8240e51124a74b81a0bdabb3ef35e66f
I20250905 19:52:16.165546 1185 ts_tablet_manager.cc:1939] T 8240e51124a74b81a0bdabb3ef35e66f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.165756 1236 raft_consensus.cc:513] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.166160 1175 raft_consensus.cc:513] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.166504 1185 tablet_replica.cc:331] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.166579 750 tablet_service.cc:1468] Processing CreateTablet for tablet 607c8a8848304491bbd4a7971f96322f (DEFAULT_TABLE table=default.Table_cb2d5de6bdc74b069d93e22c8ee74bf9 [id=f0398b74058b4ccbac71a2cb2f584a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.166721 1185 raft_consensus.cc:2241] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:16.166812 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f1d960461ef4829960baa5cca7f7765" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:16.167047 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 607c8a8848304491bbd4a7971f96322f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.167021 914 raft_consensus.cc:3058] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.167230 1236 leader_election.cc:290] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.169744 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec810e5cc1af4b9d9f6bd421072bbc77" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:16.169812 914 raft_consensus.cc:2466] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:16.169936 913 raft_consensus.cc:3058] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.170024 755 tablet_service.cc:1468] Processing CreateTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f (DEFAULT_TABLE table=default.Table_257ae7ff742645f8ba4c7ebc4cc7ccc2 [id=4044052e369843d39980a31b1654628d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.170188 986 leader_election.cc:304] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.170454 1236 raft_consensus.cc:2802] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.170483 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3ae86cb11e24d348e2f12e982f3ab9f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.170595 1236 raft_consensus.cc:695] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:16.170885 1236 consensus_queue.cc:237] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.172310 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f (DEFAULT_TABLE table=default.Table_363859b13b284790a71ced21040081e4 [id=3d169e77e70f4a059cdf41f2f3a70171]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.172690 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9d6daaa0fe43414299c032d1f4d9bb6f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.167003 1185 pending_rounds.cc:70] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:16.174328 1185 pending_rounds.cc:77] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089529443749888 op_type: NO_OP noop_request { }
I20250905 19:52:16.174492 1185 raft_consensus.cc:2887] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:16.174523 913 raft_consensus.cc:2466] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:16.174613 1185 raft_consensus.cc:2270] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.174954 712 leader_election.cc:304] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:16.175295 1203 raft_consensus.cc:2802] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.175437 1203 raft_consensus.cc:695] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:16.175493 1185 ts_tablet_manager.cc:1905] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.175578 1175 leader_election.cc:290] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.175714 1203 consensus_queue.cc:237] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.176909 744 tablet_service.cc:1468] Processing CreateTablet for tablet a884eca0e54a44dc920114e61a199689 (DEFAULT_TABLE table=default.Table_5ce48afe3ba94518bd44699d763ac7a9 [id=8c67898a816046e084be67f08f5edb71]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.177301 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a884eca0e54a44dc920114e61a199689. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.178804 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e62dbe9e48544109446323b2cc4501f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.179104 887 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf0b20630b44230b17a80cdb53322fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.178471 1184 ts_tablet_manager.cc:1918] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.179405 1184 log.cc:1199] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/cbf0b20630b44230b17a80cdb53322fa
I20250905 19:52:16.180080 1184 ts_tablet_manager.cc:1939] T cbf0b20630b44230b17a80cdb53322fa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.180085 746 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf0b20630b44230b17a80cdb53322fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.180315 748 tablet_service.cc:1515] Processing DeleteTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 with delete_type TABLET_DATA_DELETED (replaced by tablet 8adc74a357244102801b2aa8d27d9747) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.180475 747 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.181200 1186 tablet_replica.cc:331] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.181428 1186 raft_consensus.cc:2241] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.181499 1171 tablet_bootstrap.cc:492] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.181604 1186 raft_consensus.cc:2270] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.181687 1171 ts_tablet_manager.cc:1397] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20250905 19:52:16.182626 1171 raft_consensus.cc:357] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.182874 1171 raft_consensus.cc:383] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.182958 1171 raft_consensus.cc:738] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.183842 1171 consensus_queue.cc:260] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.184924 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet cbf0b20630b44230b17a80cdb53322fa (table default.Table_eb3258db08db42738115c037a4d2912d [id=3ce5d572e4104f8cb8100e115eaa8df0]) successfully deleted
I20250905 19:52:16.186154 1169 tablet_bootstrap.cc:492] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.186160 1171 ts_tablet_manager.cc:1428] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:16.186339 1169 ts_tablet_manager.cc:1397] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.051s user 0.000s sys 0.007s
I20250905 19:52:16.187219 1169 raft_consensus.cc:357] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.187390 1171 tablet_bootstrap.cc:492] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.187429 1169 raft_consensus.cc:383] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.187510 1169 raft_consensus.cc:738] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.187815 1169 consensus_queue.cc:260] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.189065 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4f1d960461ef4829960baa5cca7f7765" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:16.189368 1171 tablet_bootstrap.cc:654] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.189714 1169 ts_tablet_manager.cc:1428] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:16.191051 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ec810e5cc1af4b9d9f6bd421072bbc77" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:16.191149 1169 tablet_bootstrap.cc:492] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.191237 1053 raft_consensus.cc:3058] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.193542 1169 tablet_bootstrap.cc:654] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.194235 1053 raft_consensus.cc:2466] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
W20250905 19:52:16.194636 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 09b46a12076e492e89d0c0178bcf0524 (table default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a [id=3b8967eb30ad46cdbf272136f23318f1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 09b46a12076e492e89d0c0178bcf0524 consensus state has no leader
I20250905 19:52:16.196293 887 tablet_service.cc:1515] Processing DeleteTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.196386 1185 ts_tablet_manager.cc:1918] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:16.196552 1185 log.cc:1199] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e5a7c8a1fe604706be88d0461b241a31
I20250905 19:52:16.196614 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.197011 1185 ts_tablet_manager.cc:1939] T e5a7c8a1fe604706be88d0461b241a31 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.196966 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8240e51124a74b81a0bdabb3ef35e66f (table default.Table_d1558a276407476f80051a63eea7bdaf [id=ff84de6d110a451c9b8da5ce3803b1f2]) successfully deleted
I20250905 19:52:16.199947 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 8adc74a357244102801b2aa8d27d9747 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.200536 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_db38b4b80e674f46846361436677e883" } modify_external_catalogs: true
I20250905 19:52:16.200752 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_db38b4b80e674f46846361436677e883" } modify_external_catalogs: true
I20250905 19:52:16.200953 1169 tablet_bootstrap.cc:492] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.201104 1169 ts_tablet_manager.cc:1397] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20250905 19:52:16.201835 1169 raft_consensus.cc:357] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.201985 1169 raft_consensus.cc:383] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.202080 1169 raft_consensus.cc:738] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.202282 1169 consensus_queue.cc:260] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.202967 1169 ts_tablet_manager.cc:1428] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:16.203294 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf0b20630b44230b17a80cdb53322fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.203564 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet afb404e5b06c452bb98abeea95070a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.203806 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.204172 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.204254 1184 tablet_replica.cc:331] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.204387 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet ef9b3f29d521431f9eb8ea06ece31552 with delete_type TABLET_DATA_DELETED (replaced by tablet 8adc74a357244102801b2aa8d27d9747) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.204458 1184 raft_consensus.cc:2241] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.204586 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.204664 1184 raft_consensus.cc:2270] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.205183 1184 ts_tablet_manager.cc:1905] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:16.205296 984 leader_election.cc:343] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.208099 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.208628 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.208839 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.198668 1171 tablet_bootstrap.cc:492] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.209205 1169 tablet_bootstrap.cc:492] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.209265 1171 ts_tablet_manager.cc:1397] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.006s
I20250905 19:52:16.210258 1171 raft_consensus.cc:357] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.210517 1171 raft_consensus.cc:383] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.210598 1171 raft_consensus.cc:738] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.210888 1171 consensus_queue.cc:260] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.211530 1171 ts_tablet_manager.cc:1428] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:16.211826 1171 tablet_bootstrap.cc:492] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.213544 1171 tablet_bootstrap.cc:654] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.213599 1169 tablet_bootstrap.cc:654] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.213714 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet cbf0b20630b44230b17a80cdb53322fa because the tablet was not found. No further retry: Not found: Tablet not found: cbf0b20630b44230b17a80cdb53322fa
I20250905 19:52:16.213922 1184 ts_tablet_manager.cc:1918] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.214118 1184 log.cc:1199] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/afb404e5b06c452bb98abeea95070a5f
I20250905 19:52:16.214336 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_363859b13b284790a71ced21040081e4" } modify_external_catalogs: true
I20250905 19:52:16.214666 1184 ts_tablet_manager.cc:1939] T afb404e5b06c452bb98abeea95070a5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.215421 1185 tablet_replica.cc:331] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.215611 1185 raft_consensus.cc:2241] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:16.215862 1185 pending_rounds.cc:70] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:16.215929 1185 pending_rounds.cc:77] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089528424161280 op_type: NO_OP noop_request { }
I20250905 19:52:16.216068 1185 raft_consensus.cc:2887] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:16.216171 1185 raft_consensus.cc:2270] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.216380 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet afb404e5b06c452bb98abeea95070a5f (table default.Table_3c27b9d996a542a1bed7229a301e6a9e [id=b1e354dd48554016baae491637511fa7]) successfully deleted
I20250905 19:52:16.216920 1184 tablet_replica.cc:331] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.217265 1184 raft_consensus.cc:2241] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.217559 1184 raft_consensus.cc:2270] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.218344 1184 ts_tablet_manager.cc:1905] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.219769 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e5a7c8a1fe604706be88d0461b241a31 (table default.Table_db4bfb066c024a73b14c6c9d1f02a6cd [id=53cc2a672c8b437ea0540508da565731]) successfully deleted
I20250905 19:52:16.221076 1185 ts_tablet_manager.cc:1905] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.223280 892 tablet_service.cc:1468] Processing CreateTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 (DEFAULT_TABLE table=default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.223713 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a5c85dbda140d882a5a63a13aaabd7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:16.225243 441 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111) [suppressed 10 similar messages]
I20250905 19:52:16.225217 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 (DEFAULT_TABLE table=default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.225737 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a5c85dbda140d882a5a63a13aaabd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.235021 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c13a5b598a4597931ddf5d476aec85) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.235110 891 tablet_service.cc:1515] Processing DeleteTablet for tablet e5a7c8a1fe604706be88d0461b241a31 with delete_type TABLET_DATA_DELETED (replaced by tablet 6528897cc4b248898a44aad6a9ccaeae) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.249958 1184 ts_tablet_manager.cc:1918] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.249958 1169 tablet_bootstrap.cc:492] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.250169 1184 log.cc:1199] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8240e51124a74b81a0bdabb3ef35e66f
I20250905 19:52:16.250403 1169 ts_tablet_manager.cc:1397] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.041s user 0.018s sys 0.018s
I20250905 19:52:16.250559 1184 ts_tablet_manager.cc:1939] T 8240e51124a74b81a0bdabb3ef35e66f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.251866 1169 raft_consensus.cc:357] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.252203 1169 raft_consensus.cc:383] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.252343 1169 raft_consensus.cc:738] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.252664 1169 consensus_queue.cc:260] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.253626 1236 consensus_queue.cc:1035] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:16.253803 1185 ts_tablet_manager.cc:1918] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:16.253957 1185 log.cc:1199] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/afb404e5b06c452bb98abeea95070a5f
I20250905 19:52:16.254375 1185 ts_tablet_manager.cc:1939] T afb404e5b06c452bb98abeea95070a5f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.255359 1169 ts_tablet_manager.cc:1428] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:16.255439 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet afb404e5b06c452bb98abeea95070a5f (table default.Table_3c27b9d996a542a1bed7229a301e6a9e [id=b1e354dd48554016baae491637511fa7]) successfully deleted
I20250905 19:52:16.256677 1169 tablet_bootstrap.cc:492] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.256853 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8240e51124a74b81a0bdabb3ef35e66f (table default.Table_d1558a276407476f80051a63eea7bdaf [id=ff84de6d110a451c9b8da5ce3803b1f2]) successfully deleted
I20250905 19:52:16.258544 1236 raft_consensus.cc:491] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.258752 1236 raft_consensus.cc:513] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.259516 1236 leader_election.cc:290] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:16.260329 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d23d6df63c1a4e02a43bccf625e200bb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:16.260911 986 leader_election.cc:343] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.261029 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d23d6df63c1a4e02a43bccf625e200bb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.261514 1169 tablet_bootstrap.cc:654] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.261591 984 leader_election.cc:343] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.261734 984 leader_election.cc:304] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:16.261933 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 0480eda7e20c4336bbeacad1e7cdd56a because the tablet was not found. No further retry: Not found: Tablet not found: 0480eda7e20c4336bbeacad1e7cdd56a
I20250905 19:52:16.262439 1236 raft_consensus.cc:2747] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.262514 1184 tablet_replica.cc:331] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.262763 1184 raft_consensus.cc:2241] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.262995 1184 raft_consensus.cc:2270] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.265614 1184 ts_tablet_manager.cc:1905] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.267403 1273 raft_consensus.cc:491] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.267647 1273 raft_consensus.cc:513] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.255316 1185 tablet_replica.cc:331] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.268234 1185 raft_consensus.cc:2241] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.268447 1185 raft_consensus.cc:2270] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.268501 1273 leader_election.cc:290] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.269042 1185 ts_tablet_manager.cc:1905] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:16.269809 849 leader_election.cc:343] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: f842c6ac96ec44ffaae02edb4a9fdeef
W20250905 19:52:16.272260 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a3ae86cb11e24d348e2f12e982f3ab9f consensus state has no leader
I20250905 19:52:16.274791 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f3bcb1e148584db3b4336af7b8028437" } modify_external_catalogs: true
I20250905 19:52:16.274998 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f3bcb1e148584db3b4336af7b8028437" } modify_external_catalogs: true
I20250905 19:52:16.275401 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_39151a7d21ac4d49bb5728f518e3db4f" } modify_external_catalogs: true
I20250905 19:52:16.275550 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_39151a7d21ac4d49bb5728f518e3db4f" } modify_external_catalogs: true
W20250905 19:52:16.277908 848 leader_election.cc:343] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: f842c6ac96ec44ffaae02edb4a9fdeef
I20250905 19:52:16.278069 848 leader_election.cc:304] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.278472 1273 raft_consensus.cc:2747] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.279273 1170 tablet_bootstrap.cc:492] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:16.280182 755 tablet_service.cc:1468] Processing CreateTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 (DEFAULT_TABLE table=default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.280676 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76a5c85dbda140d882a5a63a13aaabd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.281944 1186 ts_tablet_manager.cc:1905] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.294358 1185 ts_tablet_manager.cc:1918] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.294504 1185 log.cc:1199] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ef9b3f29d521431f9eb8ea06ece31552
I20250905 19:52:16.297487 1186 ts_tablet_manager.cc:1918] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.297750 1186 log.cc:1199] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/cbf0b20630b44230b17a80cdb53322fa
I20250905 19:52:16.298454 1186 ts_tablet_manager.cc:1939] T cbf0b20630b44230b17a80cdb53322fa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.301601 1169 tablet_bootstrap.cc:492] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.301875 1169 ts_tablet_manager.cc:1397] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.019s
I20250905 19:52:16.301846 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_cb2d5de6bdc74b069d93e22c8ee74bf9" } new_table_name: "default.Table_3f876cdc8a6e4380a81a2b4d49cfb138" modify_external_catalogs: true
I20250905 19:52:16.302676 1185 ts_tablet_manager.cc:1939] T ef9b3f29d521431f9eb8ea06ece31552 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.302879 1169 raft_consensus.cc:357] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.303112 1169 raft_consensus.cc:383] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.303212 1169 raft_consensus.cc:738] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:16.303427 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85c648f799304fa9a94bec665a2dcf97 consensus state has no leader
I20250905 19:52:16.303481 1169 consensus_queue.cc:260] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.303618 1185 tablet_replica.cc:331] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.303818 1185 raft_consensus.cc:2241] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.304009 1185 raft_consensus.cc:2270] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.303862 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a5a47a603b674a41b47e7bf4b55b6a1b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.304461 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a5a47a603b674a41b47e7bf4b55b6a1b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.304525 1185 ts_tablet_manager.cc:1905] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:16.306804 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet be73873da994402e84fff6aa2ded7525 consensus state has no leader
I20250905 19:52:16.306931 1169 ts_tablet_manager.cc:1428] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:16.307215 1169 tablet_bootstrap.cc:492] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.307212 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c7b1f5de2f2f4657971e14388d57ef5a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.307890 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c7b1f5de2f2f4657971e14388d57ef5a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.308297 1176 consensus_queue.cc:1035] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:16.309016 1169 tablet_bootstrap.cc:654] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.311136 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ef9b3f29d521431f9eb8ea06ece31552 (table default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]) successfully deleted
I20250905 19:52:16.312598 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet cbf0b20630b44230b17a80cdb53322fa (table default.Table_eb3258db08db42738115c037a4d2912d [id=3ce5d572e4104f8cb8100e115eaa8df0]) successfully deleted
I20250905 19:52:16.316790 1186 tablet_replica.cc:331] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.317402 1186 raft_consensus.cc:2241] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.317507 1184 ts_tablet_manager.cc:1918] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.317675 1184 log.cc:1199] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ef9b3f29d521431f9eb8ea06ece31552
I20250905 19:52:16.318142 1184 ts_tablet_manager.cc:1939] T ef9b3f29d521431f9eb8ea06ece31552 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.318665 1169 tablet_bootstrap.cc:492] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.318840 1169 ts_tablet_manager.cc:1397] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.002s
I20250905 19:52:16.319242 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ef9b3f29d521431f9eb8ea06ece31552 (table default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]) successfully deleted
I20250905 19:52:16.319547 1184 tablet_replica.cc:331] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.319774 1184 raft_consensus.cc:2241] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.319682 1169 raft_consensus.cc:357] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.319940 1184 raft_consensus.cc:2270] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.319921 1169 raft_consensus.cc:383] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.320058 1169 raft_consensus.cc:738] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.320328 1169 consensus_queue.cc:260] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.320879 1184 ts_tablet_manager.cc:1905] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.321837 1169 ts_tablet_manager.cc:1428] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250905 19:52:16.323027 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 57f851a6eecc4a41806bdaa19cc5afab has no consensus state
I20250905 19:52:16.323068 1169 tablet_bootstrap.cc:492] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.326804 1169 tablet_bootstrap.cc:654] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.328239 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_460fbdca606041adbd01239f6cfff523"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.328876 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_460fbdca606041adbd01239f6cfff523 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.331923 1186 raft_consensus.cc:2270] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.332012 1236 raft_consensus.cc:491] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.332208 1236 raft_consensus.cc:513] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.332880 1236 leader_election.cc:290] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.333448 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aa06165136a4fa5a935dfdd786a75c4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.333665 913 raft_consensus.cc:2466] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:16.334097 986 leader_election.cc:304] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.334388 1236 raft_consensus.cc:2802] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.334503 1236 raft_consensus.cc:491] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.334630 1236 raft_consensus.cc:3058] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.332855 1170 tablet_bootstrap.cc:654] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.333454 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aa06165136a4fa5a935dfdd786a75c4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:16.335216 984 leader_election.cc:343] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.337613 1236 raft_consensus.cc:513] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.338704 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aa06165136a4fa5a935dfdd786a75c4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:16.338896 914 raft_consensus.cc:3058] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.340229 1184 ts_tablet_manager.cc:1918] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.340400 1184 log.cc:1199] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e5a7c8a1fe604706be88d0461b241a31
I20250905 19:52:16.340855 1184 ts_tablet_manager.cc:1939] T e5a7c8a1fe604706be88d0461b241a31 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.341467 914 raft_consensus.cc:2466] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:16.341861 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 9d6daaa0fe43414299c032d1f4d9bb6f
I20250905 19:52:16.341925 986 leader_election.cc:304] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.342007 1184 tablet_replica.cc:331] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.342186 1176 raft_consensus.cc:2802] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.342372 1176 raft_consensus.cc:695] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:16.342456 1236 leader_election.cc:290] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.342701 1176 consensus_queue.cc:237] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.343999 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e5a7c8a1fe604706be88d0461b241a31 (table default.Table_db4bfb066c024a73b14c6c9d1f02a6cd [id=53cc2a672c8b437ea0540508da565731]) successfully deleted
I20250905 19:52:16.346915 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.347728 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_702bc880676a4ad3b61ef0ec4051a032"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.348706 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:16.349011 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_702bc880676a4ad3b61ef0ec4051a032 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.350991 1273 raft_consensus.cc:491] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.351181 1273 raft_consensus.cc:513] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.351899 1273 leader_election.cc:290] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.352010 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7aa06165136a4fa5a935dfdd786a75c4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:16.353084 1184 raft_consensus.cc:2241] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.353295 1184 raft_consensus.cc:2270] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:16.353456 984 leader_election.cc:343] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.353780 1169 tablet_bootstrap.cc:492] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.353860 1184 ts_tablet_manager.cc:1905] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.353925 1169 ts_tablet_manager.cc:1397] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.031s user 0.009s sys 0.016s
I20250905 19:52:16.354491 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78ac5c7e9b6741d79db2e606d55a3509" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:16.354763 1053 raft_consensus.cc:2466] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:16.354804 1169 raft_consensus.cc:357] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.355216 849 leader_election.cc:304] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.355604 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78ac5c7e9b6741d79db2e606d55a3509" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.355674 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.355993 1273 raft_consensus.cc:2802] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.356101 1273 raft_consensus.cc:491] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.356261 1273 raft_consensus.cc:3058] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:16.356235 848 leader_election.cc:343] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.356783 1186 ts_tablet_manager.cc:1905] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.359084 1273 raft_consensus.cc:513] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.360131 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78ac5c7e9b6741d79db2e606d55a3509" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:16.360641 848 leader_election.cc:343] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.361102 1273 leader_election.cc:290] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.361271 1169 raft_consensus.cc:383] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.361384 1169 raft_consensus.cc:738] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.361622 1169 consensus_queue.cc:260] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.362919 1169 ts_tablet_manager.cc:1428] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250905 19:52:16.363219 1169 tablet_bootstrap.cc:492] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.364959 1169 tablet_bootstrap.cc:654] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.368947 1185 ts_tablet_manager.cc:1918] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.369122 1185 log.cc:1199] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/f842c6ac96ec44ffaae02edb4a9fdeef
I20250905 19:52:16.369462 1185 ts_tablet_manager.cc:1939] T f842c6ac96ec44ffaae02edb4a9fdeef P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.370555 1185 tablet_replica.cc:331] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.370738 1185 raft_consensus.cc:2241] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.370898 1185 raft_consensus.cc:2270] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.371433 1185 ts_tablet_manager.cc:1905] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.374855 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet f842c6ac96ec44ffaae02edb4a9fdeef (table default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
I20250905 19:52:16.375434 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78ac5c7e9b6741d79db2e606d55a3509" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:16.375684 1053 raft_consensus.cc:3058] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.378351 1176 raft_consensus.cc:491] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.378536 1176 raft_consensus.cc:513] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.379376 1176 leader_election.cc:290] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.380132 1053 raft_consensus.cc:2466] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:16.380146 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c6295c8c7874df89662b8cef6459fec" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.381345 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c6295c8c7874df89662b8cef6459fec" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:16.380883 984 leader_election.cc:343] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:16.383231 986 leader_election.cc:343] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.383704 986 leader_election.cc:304] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.385210 1185 ts_tablet_manager.cc:1918] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.385354 1185 log.cc:1199] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/7e62dbe9e48544109446323b2cc4501f
I20250905 19:52:16.385524 1184 ts_tablet_manager.cc:1918] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.385689 1184 log.cc:1199] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/eacedd2e38d64d2cb25b2ebd88f563a1
I20250905 19:52:16.385741 1185 ts_tablet_manager.cc:1939] T 7e62dbe9e48544109446323b2cc4501f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.386124 1176 raft_consensus.cc:2747] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.386478 1184 ts_tablet_manager.cc:1939] T eacedd2e38d64d2cb25b2ebd88f563a1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.387403 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 7e62dbe9e48544109446323b2cc4501f (table default.Table_3863927d1ffa41289c4ccd14dcec05f9 [id=567a4196a6af4e2b828347357c44313c]) successfully deleted
I20250905 19:52:16.387907 849 leader_election.cc:304] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.388350 1273 raft_consensus.cc:2802] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.388484 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet eacedd2e38d64d2cb25b2ebd88f563a1 (table default.Table_056e9cf230e44fc5badd0c9b709c0c4f [id=88e8e8eaaefb4045b64fd8ad64f9990c]) successfully deleted
I20250905 19:52:16.388551 1273 raft_consensus.cc:695] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:16.388813 1273 consensus_queue.cc:237] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.389077 1184 tablet_replica.cc:331] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.389309 1184 raft_consensus.cc:2241] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.389521 1184 raft_consensus.cc:2270] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.390182 1184 ts_tablet_manager.cc:1905] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.390755 1171 tablet_bootstrap.cc:492] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.390933 1171 ts_tablet_manager.cc:1397] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.179s user 0.024s sys 0.008s
I20250905 19:52:16.391749 1171 raft_consensus.cc:357] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.391988 1171 raft_consensus.cc:383] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.392081 1171 raft_consensus.cc:738] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.392334 1171 consensus_queue.cc:260] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.392962 1171 ts_tablet_manager.cc:1428] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:16.393291 1171 tablet_bootstrap.cc:492] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.395251 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet f8d3cbdda0c94e139259d90919dc2f95
I20250905 19:52:16.396070 755 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.397012 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.398006 892 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.399345 1171 tablet_bootstrap.cc:654] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.399662 1184 ts_tablet_manager.cc:1918] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.400204 1184 log.cc:1199] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6d09371944644b3082268b2b37ce54f8
I20250905 19:52:16.400738 1184 ts_tablet_manager.cc:1939] T 6d09371944644b3082268b2b37ce54f8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.402102 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6d09371944644b3082268b2b37ce54f8 (table default.Table_1eaa6c2091ee4d949f145d0e9e806550 [id=43eb792904d2448da31df6a56c647397]) successfully deleted
I20250905 19:52:16.402590 1184 tablet_replica.cc:331] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.402835 1184 raft_consensus.cc:2241] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:16.403163 1184 pending_rounds.cc:70] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:16.403291 1184 pending_rounds.cc:77] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089530558279680 op_type: NO_OP noop_request { }
I20250905 19:52:16.403513 1184 raft_consensus.cc:2887] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:16.403692 1184 raft_consensus.cc:2270] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.404803 1184 ts_tablet_manager.cc:1905] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.405004 1185 tablet_replica.cc:331] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.405857 1185 raft_consensus.cc:2241] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:16.406111 1185 pending_rounds.cc:70] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:16.406191 1185 pending_rounds.cc:77] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089528760250368 op_type: NO_OP noop_request { }
I20250905 19:52:16.406308 1185 raft_consensus.cc:2887] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:16.406414 1185 raft_consensus.cc:2270] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.409219 1185 ts_tablet_manager.cc:1905] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:16.411492 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 607c8a8848304491bbd4a7971f96322f consensus state has no leader
I20250905 19:52:16.414362 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6f6a2b47de6b47f0a5eeb3d53a2addd2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.414968 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6f6a2b47de6b47f0a5eeb3d53a2addd2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.417207 1171 tablet_bootstrap.cc:492] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.417383 1171 ts_tablet_manager.cc:1397] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.004s
I20250905 19:52:16.418287 1171 raft_consensus.cc:357] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.418498 1171 raft_consensus.cc:383] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.418606 1171 raft_consensus.cc:738] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.418853 1171 consensus_queue.cc:260] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.419697 1171 ts_tablet_manager.cc:1428] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:16.420024 1171 tablet_bootstrap.cc:492] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.421407 1184 ts_tablet_manager.cc:1918] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:16.421571 1184 log.cc:1199] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:16.422029 1184 ts_tablet_manager.cc:1939] T 4f1d960461ef4829960baa5cca7f7765 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.422322 1171 tablet_bootstrap.cc:654] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.423359 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 4f1d960461ef4829960baa5cca7f7765 (table default.Table_b550173c58174bedacdcb184e73e8e9c [id=c56b6d4d3b204b0981bb9cecd8a1a08c]) successfully deleted
W20250905 19:52:16.423616 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 9d6daaa0fe43414299c032d1f4d9bb6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d6daaa0fe43414299c032d1f4d9bb6f already in progress: creating tablet
W20250905 19:52:16.424043 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet f8d3cbdda0c94e139259d90919dc2f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d3cbdda0c94e139259d90919dc2f95 already in progress: creating tablet
I20250905 19:52:16.425202 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_00ad894476894e1782714d41f84589b3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.425717 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00ad894476894e1782714d41f84589b3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.427248 1185 ts_tablet_manager.cc:1918] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:16.428933 1185 log.cc:1199] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/cbf0b20630b44230b17a80cdb53322fa
I20250905 19:52:16.429514 1185 ts_tablet_manager.cc:1939] T cbf0b20630b44230b17a80cdb53322fa P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.430708 1185 tablet_replica.cc:331] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.430884 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet cbf0b20630b44230b17a80cdb53322fa (table default.Table_eb3258db08db42738115c037a4d2912d [id=3ce5d572e4104f8cb8100e115eaa8df0]) successfully deleted
I20250905 19:52:16.431674 1185 raft_consensus.cc:2241] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.431842 1185 raft_consensus.cc:2270] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.432417 1185 ts_tablet_manager.cc:1905] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.433890 1169 tablet_bootstrap.cc:492] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.434121 1169 ts_tablet_manager.cc:1397] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.016s
I20250905 19:52:16.435149 1169 raft_consensus.cc:357] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.435492 1169 raft_consensus.cc:383] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.435606 1169 raft_consensus.cc:738] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.435984 1169 consensus_queue.cc:260] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.437575 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ae188650e4914c02837028a3645a3b48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.438148 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae188650e4914c02837028a3645a3b48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.439162 1176 raft_consensus.cc:491] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.439291 1176 raft_consensus.cc:513] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.441316 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.443017 1170 tablet_bootstrap.cc:492] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:16.443195 1170 ts_tablet_manager.cc:1397] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.165s user 0.001s sys 0.003s
I20250905 19:52:16.444070 1170 raft_consensus.cc:357] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.444252 1170 raft_consensus.cc:383] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.444329 1170 raft_consensus.cc:738] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.444583 1170 consensus_queue.cc:260] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.446324 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.447211 1170 ts_tablet_manager.cc:1428] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250905 19:52:16.447278 986 leader_election.cc:343] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.448985 1170 tablet_bootstrap.cc:492] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:16.449229 984 leader_election.cc:343] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.450577 1176 leader_election.cc:290] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.450796 984 leader_election.cc:304] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.451442 1176 raft_consensus.cc:2747] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.452337 1169 ts_tablet_manager.cc:1428] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:16.454519 1169 tablet_bootstrap.cc:492] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.457798 1169 tablet_bootstrap.cc:654] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.462293 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_528007b5c2c544e7a81752f8b473d78c" } modify_external_catalogs: true
I20250905 19:52:16.462517 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_528007b5c2c544e7a81752f8b473d78c" } modify_external_catalogs: true
I20250905 19:52:16.465309 1171 tablet_bootstrap.cc:492] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.465546 1171 ts_tablet_manager.cc:1397] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.046s user 0.017s sys 0.000s
I20250905 19:52:16.466312 1185 ts_tablet_manager.cc:1918] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.466526 1171 raft_consensus.cc:357] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.466742 1171 raft_consensus.cc:383] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.466827 1171 raft_consensus.cc:738] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.466966 1185 log.cc:1199] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/eacedd2e38d64d2cb25b2ebd88f563a1
I20250905 19:52:16.467195 1170 tablet_bootstrap.cc:654] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.467622 1185 ts_tablet_manager.cc:1939] T eacedd2e38d64d2cb25b2ebd88f563a1 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.468952 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet eacedd2e38d64d2cb25b2ebd88f563a1 (table default.Table_056e9cf230e44fc5badd0c9b709c0c4f [id=88e8e8eaaefb4045b64fd8ad64f9990c]) successfully deleted
I20250905 19:52:16.470259 1185 tablet_replica.cc:331] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.470723 1185 raft_consensus.cc:2241] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.471349 1185 raft_consensus.cc:2270] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.472393 1185 ts_tablet_manager.cc:1905] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.473201 1169 tablet_bootstrap.cc:492] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.473223 449 master_service.cc:948] ReplaceTablet: received request to replace tablet ec810e5cc1af4b9d9f6bd421072bbc77 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.473428 1169 ts_tablet_manager.cc:1397] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20250905 19:52:16.474678 1169 raft_consensus.cc:357] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.474932 1169 raft_consensus.cc:383] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.475060 1169 raft_consensus.cc:738] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.475378 1169 consensus_queue.cc:260] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.478338 1169 ts_tablet_manager.cc:1428] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:16.478698 1169 tablet_bootstrap.cc:492] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.479607 1186 ts_tablet_manager.cc:1918] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.479739 1186 log.cc:1199] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ef9b3f29d521431f9eb8ea06ece31552
I20250905 19:52:16.480154 1186 ts_tablet_manager.cc:1939] T ef9b3f29d521431f9eb8ea06ece31552 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.481191 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ef9b3f29d521431f9eb8ea06ece31552 (table default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]) successfully deleted
I20250905 19:52:16.484376 1169 tablet_bootstrap.cc:654] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.488483 892 tablet_service.cc:1468] Processing CreateTablet for tablet 7c5807c1132747e6af79d93e7496d500 (DEFAULT_TABLE table=default.Table_f3bcb1e148584db3b4336af7b8028437 [id=cd492ea5b4b4434b960ba37fcf479450]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.488938 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c5807c1132747e6af79d93e7496d500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.493460 454 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 1c6295c8c7874df89662b8cef6459fec (table default.Table_0de80dfdff2546a2a39368585f13a394 [id=2151f6f3348c4d1fb376448ecfcbae8a]). Expected version 1 got 0
I20250905 19:52:16.493788 454 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d23d6df63c1a4e02a43bccf625e200bb (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]). Expected version 1 got 0
I20250905 19:52:16.494163 893 tablet_service.cc:1468] Processing CreateTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab (DEFAULT_TABLE table=default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.494122 454 catalog_manager.cc:5582] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:16.494686 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f851a6eecc4a41806bdaa19cc5afab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.494900 890 tablet_service.cc:1468] Processing CreateTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.495239 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d12c7fbb84849c68274810b3fdcba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.496269 894 tablet_service.cc:1468] Processing CreateTablet for tablet e6f837fb7293474b8ad357f52e42e42e (DEFAULT_TABLE table=default.Table_1fd86ae5263542ed92214968ec16ccf0 [id=02ab99d2ef0f499db0d3625ede6c6e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.496636 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6f837fb7293474b8ad357f52e42e42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.497449 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 78ac5c7e9b6741d79db2e606d55a3509 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.498028 455 catalog_manager.cc:5582] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:16.498375 886 tablet_service.cc:1468] Processing CreateTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 (DEFAULT_TABLE table=default.Table_ff5415e10957422498084d7730365989 [id=bd0b3b55bf3648b7a45b7aef904be570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.498486 455 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_edca9fb5f8fb48d28540ecd7e385178c [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 3 got 0
I20250905 19:52:16.498730 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e4edddafcd4d22a0e0ae66b0b953e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.500264 888 tablet_service.cc:1468] Processing CreateTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c (DEFAULT_TABLE table=default.Table_39151a7d21ac4d49bb5728f518e3db4f [id=e0a1acd20bc340088714693c74f31d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.500636 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77cbdda0e2d641bd8092f00925bb9a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.501711 755 tablet_service.cc:1468] Processing CreateTablet for tablet 7c5807c1132747e6af79d93e7496d500 (DEFAULT_TABLE table=default.Table_f3bcb1e148584db3b4336af7b8028437 [id=cd492ea5b4b4434b960ba37fcf479450]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.502120 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c5807c1132747e6af79d93e7496d500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.504256 887 tablet_service.cc:1468] Processing CreateTablet for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (DEFAULT_TABLE table=default.Table_feeb076a5fc84dd38f2fa2908e15b6b7 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.504292 750 tablet_service.cc:1468] Processing CreateTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab (DEFAULT_TABLE table=default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.504503 745 tablet_service.cc:1468] Processing CreateTablet for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (DEFAULT_TABLE table=default.Table_feeb076a5fc84dd38f2fa2908e15b6b7 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.504603 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.504652 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f851a6eecc4a41806bdaa19cc5afab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.504892 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.508226 891 tablet_service.cc:1468] Processing CreateTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 (DEFAULT_TABLE table=default.Table_b86b0c6f5fe4496ead402827d2db3f1d [id=8638b095c62741559b2281befee964ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.508275 747 tablet_service.cc:1468] Processing CreateTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c (DEFAULT_TABLE table=default.Table_39151a7d21ac4d49bb5728f518e3db4f [id=e0a1acd20bc340088714693c74f31d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.508700 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77cbdda0e2d641bd8092f00925bb9a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.509826 1170 tablet_bootstrap.cc:492] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:16.510022 1170 ts_tablet_manager.cc:1397] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.000s
I20250905 19:52:16.511533 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.511562 1170 raft_consensus.cc:357] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.511785 1170 raft_consensus.cc:383] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.511873 1170 raft_consensus.cc:738] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.512297 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.498199 1176 raft_consensus.cc:491] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.513516 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.513468 1176 raft_consensus.cc:513] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.513931 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d12c7fbb84849c68274810b3fdcba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.514269 1176 leader_election.cc:290] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:16.506242 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab (DEFAULT_TABLE table=default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.515206 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 57f851a6eecc4a41806bdaa19cc5afab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.516875 748 tablet_service.cc:1468] Processing CreateTablet for tablet e6f837fb7293474b8ad357f52e42e42e (DEFAULT_TABLE table=default.Table_1fd86ae5263542ed92214968ec16ccf0 [id=02ab99d2ef0f499db0d3625ede6c6e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.517300 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6f837fb7293474b8ad357f52e42e42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.506615 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 (DEFAULT_TABLE table=default.Table_b86b0c6f5fe4496ead402827d2db3f1d [id=8638b095c62741559b2281befee964ea]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.518708 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.506925 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (DEFAULT_TABLE table=default.Table_feeb076a5fc84dd38f2fa2908e15b6b7 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.524415 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.509156 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 7c5807c1132747e6af79d93e7496d500 (DEFAULT_TABLE table=default.Table_f3bcb1e148584db3b4336af7b8028437 [id=cd492ea5b4b4434b960ba37fcf479450]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.528398 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7c5807c1132747e6af79d93e7496d500. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.509491 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 (DEFAULT_TABLE table=default.Table_ff5415e10957422498084d7730365989 [id=bd0b3b55bf3648b7a45b7aef904be570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.529243 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e4edddafcd4d22a0e0ae66b0b953e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.512835 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c (DEFAULT_TABLE table=default.Table_39151a7d21ac4d49bb5728f518e3db4f [id=e0a1acd20bc340088714693c74f31d2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.530128 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 77cbdda0e2d641bd8092f00925bb9a9c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.513173 1024 tablet_service.cc:1468] Processing CreateTablet for tablet e6f837fb7293474b8ad357f52e42e42e (DEFAULT_TABLE table=default.Table_1fd86ae5263542ed92214968ec16ccf0 [id=02ab99d2ef0f499db0d3625ede6c6e6e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.530771 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6f837fb7293474b8ad357f52e42e42e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.532094 754 tablet_service.cc:1468] Processing CreateTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 (DEFAULT_TABLE table=default.Table_ff5415e10957422498084d7730365989 [id=bd0b3b55bf3648b7a45b7aef904be570]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.533006 1170 consensus_queue.cc:260] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.533427 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 59c13a5b598a4597931ddf5d476aec85 (DEFAULT_TABLE table=default.Table_755dce6406b647a1b2fffb9aa254e403 [id=5ac15645cb4d4730a4b4464a85299bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.533905 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c13a5b598a4597931ddf5d476aec85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.540551 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "305a7758527d4d52a669b2f2c4ceedbb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.540692 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 57259d8eb0d44d1cbac80047c7ad67b6 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.541743 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.504289 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 (DEFAULT_TABLE table=default.Table_b86b0c6f5fe4496ead402827d2db3f1d [id=8638b095c62741559b2281befee964ea]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:16.542893 984 leader_election.cc:343] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.551131 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 19e4edddafcd4d22a0e0ae66b0b953e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.555291 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.556253 753 tablet_service.cc:1468] Processing CreateTablet for tablet 59c13a5b598a4597931ddf5d476aec85 (DEFAULT_TABLE table=default.Table_755dce6406b647a1b2fffb9aa254e403 [id=5ac15645cb4d4730a4b4464a85299bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.556706 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c13a5b598a4597931ddf5d476aec85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.558259 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6d12c7fbb84849c68274810b3fdcba04. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.558840 1171 consensus_queue.cc:260] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.559401 1171 ts_tablet_manager.cc:1428] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.094s user 0.001s sys 0.001s
I20250905 19:52:16.560783 1171 tablet_bootstrap.cc:492] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.564774 1171 tablet_bootstrap.cc:654] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.567682 888 tablet_service.cc:1468] Processing CreateTablet for tablet 59c13a5b598a4597931ddf5d476aec85 (DEFAULT_TABLE table=default.Table_755dce6406b647a1b2fffb9aa254e403 [id=5ac15645cb4d4730a4b4464a85299bed]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.568145 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 59c13a5b598a4597931ddf5d476aec85. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.576844 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "305a7758527d4d52a669b2f2c4ceedbb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.577553 1186 tablet_replica.cc:331] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:16.581010 986 leader_election.cc:343] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.581174 986 leader_election.cc:304] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.581691 1176 raft_consensus.cc:2747] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.581854 1185 ts_tablet_manager.cc:1918] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.581979 1185 log.cc:1199] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:16.582345 1185 ts_tablet_manager.cc:1939] T 4f1d960461ef4829960baa5cca7f7765 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.583307 1185 tablet_replica.cc:331] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.583539 1185 raft_consensus.cc:2241] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:16.587407 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 4f1d960461ef4829960baa5cca7f7765 (table default.Table_b550173c58174bedacdcb184e73e8e9c [id=c56b6d4d3b204b0981bb9cecd8a1a08c]) successfully deleted
I20250905 19:52:16.587886 1203 raft_consensus.cc:491] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.592427 1203 raft_consensus.cc:513] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.593179 1203 leader_election.cc:290] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:16.594715 711 leader_election.cc:343] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: eacedd2e38d64d2cb25b2ebd88f563a1
W20250905 19:52:16.595184 712 leader_election.cc:343] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: eacedd2e38d64d2cb25b2ebd88f563a1
I20250905 19:52:16.595268 1185 pending_rounds.cc:70] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:16.595284 712 leader_election.cc:304] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:16.595363 1185 pending_rounds.cc:77] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089531451129856 op_type: NO_OP noop_request { }
I20250905 19:52:16.595467 1185 raft_consensus.cc:2887] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:16.595584 1185 raft_consensus.cc:2270] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.596153 1203 raft_consensus.cc:2747] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.596484 1185 ts_tablet_manager.cc:1905] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.599004 448 master_service.cc:948] ReplaceTablet: received request to replace tablet cbf0b20630b44230b17a80cdb53322fa from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.599396 446 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c7eeb8109f3b4be885a859d6c4e90061
I20250905 19:52:16.599713 446 master_service.cc:948] ReplaceTablet: received request to replace tablet a3ae86cb11e24d348e2f12e982f3ab9f from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.600190 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 7c5807c1132747e6af79d93e7496d500
I20250905 19:52:16.601521 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c5807c1132747e6af79d93e7496d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:16.602243 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 7c5807c1132747e6af79d93e7496d500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c5807c1132747e6af79d93e7496d500 already in progress: creating tablet
W20250905 19:52:16.604013 454 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d23d6df63c1a4e02a43bccf625e200bb (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d23d6df63c1a4e02a43bccf625e200bb consensus state has no leader
I20250905 19:52:16.605118 1169 tablet_bootstrap.cc:492] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.605283 1169 ts_tablet_manager.cc:1397] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.127s user 0.024s sys 0.014s
I20250905 19:52:16.606738 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8adc74a357244102801b2aa8d27d9747 with delete_type TABLET_DATA_DELETED (replaced by tablet 2201e4bad99643a9b041f6f99406433b) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.606865 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet b776063b81c44217b517a619af927c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.606760 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.607041 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.607170 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c13a5b598a4597931ddf5d476aec85) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.607452 1184 tablet_replica.cc:331] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.607646 1184 raft_consensus.cc:2241] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.607863 1184 raft_consensus.cc:2270] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.607646 1169 raft_consensus.cc:357] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.608381 1169 raft_consensus.cc:383] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.608460 1169 raft_consensus.cc:738] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.608722 1184 ts_tablet_manager.cc:1905] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.608714 1169 consensus_queue.cc:260] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.609392 1169 ts_tablet_manager.cc:1428] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:16.609944 1169 tablet_bootstrap.cc:492] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.611104 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c5807c1132747e6af79d93e7496d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.612772 1169 tablet_bootstrap.cc:654] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.612942 447 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:16.615551 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.616237 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b776063b81c44217b517a619af927c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.618383 1171 tablet_bootstrap.cc:492] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.618533 1171 ts_tablet_manager.cc:1397] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.059s user 0.002s sys 0.006s
I20250905 19:52:16.619343 1186 raft_consensus.cc:2241] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.619586 1186 raft_consensus.cc:2270] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.619556 1171 raft_consensus.cc:357] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.619764 1171 raft_consensus.cc:383] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.619870 1171 raft_consensus.cc:738] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.620489 1186 ts_tablet_manager.cc:1905] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.623682 1184 ts_tablet_manager.cc:1918] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.623809 1184 log.cc:1199] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8adc74a357244102801b2aa8d27d9747
I20250905 19:52:16.624131 1171 consensus_queue.cc:260] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.624430 1169 tablet_bootstrap.cc:492] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.624617 1169 ts_tablet_manager.cc:1397] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.002s
I20250905 19:52:16.625495 1169 raft_consensus.cc:357] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.625757 1169 raft_consensus.cc:383] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.625900 1169 raft_consensus.cc:738] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.626632 1171 ts_tablet_manager.cc:1428] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250905 19:52:16.627853 1170 ts_tablet_manager.cc:1428] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.086s user 0.003s sys 0.000s
I20250905 19:52:16.627266 1169 consensus_queue.cc:260] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.629957 1169 ts_tablet_manager.cc:1428] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:16.630291 1169 tablet_bootstrap.cc:492] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.632550 1169 tablet_bootstrap.cc:654] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.637598 1171 tablet_bootstrap.cc:492] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.640103 1171 tablet_bootstrap.cc:654] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.651468 1184 ts_tablet_manager.cc:1939] T 8adc74a357244102801b2aa8d27d9747 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.651826 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c5807c1132747e6af79d93e7496d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.653007 1184 tablet_replica.cc:331] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.653187 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8adc74a357244102801b2aa8d27d9747 (table default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]) successfully deleted
I20250905 19:52:16.653350 1184 raft_consensus.cc:2241] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.653939 1184 raft_consensus.cc:2270] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.654771 1184 ts_tablet_manager.cc:1905] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.655525 1185 ts_tablet_manager.cc:1918] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:16.656273 1185 log.cc:1199] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/78ac5c7e9b6741d79db2e606d55a3509
I20250905 19:52:16.656912 1185 ts_tablet_manager.cc:1939] T 78ac5c7e9b6741d79db2e606d55a3509 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.659575 1186 ts_tablet_manager.cc:1918] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.659724 1186 log.cc:1199] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e5a7c8a1fe604706be88d0461b241a31
I20250905 19:52:16.660343 1170 tablet_bootstrap.cc:492] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:16.661650 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 78ac5c7e9b6741d79db2e606d55a3509 (table default.Table_755dce6406b647a1b2fffb9aa254e403 [id=5ac15645cb4d4730a4b4464a85299bed]) successfully deleted
I20250905 19:52:16.662418 1170 tablet_bootstrap.cc:654] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.662614 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet e5a7c8a1fe604706be88d0461b241a31 because the tablet was not found. No further retry: Not found: Tablet not found: e5a7c8a1fe604706be88d0461b241a31
I20250905 19:52:16.663908 1171 tablet_bootstrap.cc:492] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.664234 1171 ts_tablet_manager.cc:1397] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.027s user 0.001s sys 0.004s
I20250905 19:52:16.665318 1171 raft_consensus.cc:357] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.665678 1171 raft_consensus.cc:383] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.665817 1171 raft_consensus.cc:738] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.666239 1171 consensus_queue.cc:260] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:16.667243 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 9d6daaa0fe43414299c032d1f4d9bb6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d6daaa0fe43414299c032d1f4d9bb6f already in progress: creating tablet
W20250905 19:52:16.673650 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet f8d3cbdda0c94e139259d90919dc2f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d3cbdda0c94e139259d90919dc2f95 already in progress: creating tablet
I20250905 19:52:16.675135 1171 ts_tablet_manager.cc:1428] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:16.675488 1185 tablet_replica.cc:331] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.675910 1185 raft_consensus.cc:2241] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.678957 1186 ts_tablet_manager.cc:1939] T e5a7c8a1fe604706be88d0461b241a31 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.682785 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet e5a7c8a1fe604706be88d0461b241a31 (table default.Table_db4bfb066c024a73b14c6c9d1f02a6cd [id=53cc2a672c8b437ea0540508da565731]) successfully deleted
W20250905 19:52:16.685503 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9d6daaa0fe43414299c032d1f4d9bb6f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9d6daaa0fe43414299c032d1f4d9bb6f already in progress: creating tablet
W20250905 19:52:16.687054 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet f8d3cbdda0c94e139259d90919dc2f95 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f8d3cbdda0c94e139259d90919dc2f95 already in progress: creating tablet
I20250905 19:52:16.687214 1184 ts_tablet_manager.cc:1918] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.687920 1185 raft_consensus.cc:2270] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:16.688360 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 7c5807c1132747e6af79d93e7496d500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c5807c1132747e6af79d93e7496d500 already in progress: creating tablet
I20250905 19:52:16.687691 1171 tablet_bootstrap.cc:492] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.691551 1171 tablet_bootstrap.cc:654] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.694852 1185 ts_tablet_manager.cc:1905] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.695647 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 77cbdda0e2d641bd8092f00925bb9a9c
I20250905 19:52:16.697189 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.699905 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.702494 746 tablet_service.cc:1515] Processing DeleteTablet for tablet afb404e5b06c452bb98abeea95070a5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.702826 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7e62dbe9e48544109446323b2cc4501f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.703461 1185 ts_tablet_manager.cc:1918] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.703655 1185 log.cc:1199] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6d09371944644b3082268b2b37ce54f8
I20250905 19:52:16.698237 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.699470 1169 tablet_bootstrap.cc:492] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.704107 1185 ts_tablet_manager.cc:1939] T 6d09371944644b3082268b2b37ce54f8 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.704064 754 tablet_service.cc:1515] Processing DeleteTablet for tablet cbf0b20630b44230b17a80cdb53322fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.704321 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 8240e51124a74b81a0bdabb3ef35e66f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.704332 1169 ts_tablet_manager.cc:1397] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.074s user 0.031s sys 0.029s
I20250905 19:52:16.705307 1185 tablet_replica.cc:331] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.705188 1169 raft_consensus.cc:357] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.705317 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6d09371944644b3082268b2b37ce54f8 (table default.Table_1eaa6c2091ee4d949f145d0e9e806550 [id=43eb792904d2448da31df6a56c647397]) successfully deleted
I20250905 19:52:16.705403 1169 raft_consensus.cc:383] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.705469 1169 raft_consensus.cc:738] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.705538 1185 raft_consensus.cc:2241] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.705744 1185 raft_consensus.cc:2270] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.705710 1169 consensus_queue.cc:260] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:16.705942 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 7c5807c1132747e6af79d93e7496d500 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7c5807c1132747e6af79d93e7496d500 already in progress: creating tablet
I20250905 19:52:16.706557 1169 ts_tablet_manager.cc:1428] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:16.706885 1169 tablet_bootstrap.cc:492] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.707413 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:16.708529 453 catalog_manager.cc:6652] ReplaceTablet: tablet 57259d8eb0d44d1cbac80047c7ad67b6 deleted and replaced by tablet 01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:16.708730 1186 tablet_replica.cc:331] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.708917 1169 tablet_bootstrap.cc:654] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.708914 1186 raft_consensus.cc:2241] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.709090 1186 raft_consensus.cc:2270] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.709817 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c20eea1b024d7f93f18f0c089efc42) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.709726 1186 ts_tablet_manager.cc:1905] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.710651 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c20eea1b024d7f93f18f0c089efc42) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:16.712214 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 77cbdda0e2d641bd8092f00925bb9a9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77cbdda0e2d641bd8092f00925bb9a9c already in progress: creating tablet
I20250905 19:52:16.714042 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c20eea1b024d7f93f18f0c089efc42) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.714887 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_07f86749b59347fbb1060d7102299913"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.715456 1185 ts_tablet_manager.cc:1905] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.719398 1184 log.cc:1199] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/b776063b81c44217b517a619af927c08
I20250905 19:52:16.719926 1184 ts_tablet_manager.cc:1939] T b776063b81c44217b517a619af927c08 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.722234 1186 ts_tablet_manager.cc:1918] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.722393 1184 tablet_replica.cc:331] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.722452 1186 log.cc:1199] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/afb404e5b06c452bb98abeea95070a5f
I20250905 19:52:16.722786 1184 raft_consensus.cc:2241] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.722868 1186 ts_tablet_manager.cc:1939] T afb404e5b06c452bb98abeea95070a5f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.723104 1184 raft_consensus.cc:2270] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.723815 1186 tablet_replica.cc:331] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.723944 1184 ts_tablet_manager.cc:1905] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.724102 1186 raft_consensus.cc:2241] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.724334 1186 raft_consensus.cc:2270] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:16.725987 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07f86749b59347fbb1060d7102299913 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.715916 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c9d143854de148aea480b85f2718ef23" } new_table_name: "default.Table_c9fd3a3508c14b0b8d403de08fde0b82" modify_external_catalogs: true
I20250905 19:52:16.717916 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6eb260fb20e14dc59c7f4bdb683fa538" } new_table_name: "default.Table_084f2c1b9d144e38af5948cfd007d374" modify_external_catalogs: true
I20250905 19:52:16.732594 1186 ts_tablet_manager.cc:1905] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.719784 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ccbfb18fea374ff289a19816635f3afd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.733610 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ccbfb18fea374ff289a19816635f3afd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.734355 1184 ts_tablet_manager.cc:1918] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.720892 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_97c63bdbe93a48708207c1e9a96cb1bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.721385 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet b776063b81c44217b517a619af927c08 (table default.Table_ce8e455023774d02ba896de13623c0ea [id=06e2f51841544340adc3a39ca41a9bea]) successfully deleted
W20250905 19:52:16.734985 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97c63bdbe93a48708207c1e9a96cb1bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:16.735073 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 4f1d960461ef4829960baa5cca7f7765 because the tablet was not found. No further retry: Not found: Tablet not found: 4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:16.735469 1184 log.cc:1199] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6f4ec474911941b2a2d0825cdb37bd72
I20250905 19:52:16.736197 1169 tablet_bootstrap.cc:492] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.736343 1169 ts_tablet_manager.cc:1397] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.030s user 0.015s sys 0.011s
I20250905 19:52:16.736979 1169 raft_consensus.cc:357] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.737131 1169 raft_consensus.cc:383] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.737223 1169 raft_consensus.cc:738] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.737454 1169 consensus_queue.cc:260] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.737993 1169 ts_tablet_manager.cc:1428] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:16.738255 1169 tablet_bootstrap.cc:492] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.722240 453 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:16.739236 1186 ts_tablet_manager.cc:1918] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.739405 1186 log.cc:1199] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/7e62dbe9e48544109446323b2cc4501f
I20250905 19:52:16.739758 1186 ts_tablet_manager.cc:1939] T 7e62dbe9e48544109446323b2cc4501f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.740208 1169 tablet_bootstrap.cc:654] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.740325 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_feeb076a5fc84dd38f2fa2908e15b6b7" } new_table_name: "default.Table_8c4a8cf3d2354147a7c24ecab6f27c05" modify_external_catalogs: true
I20250905 19:52:16.722529 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_edca9fb5f8fb48d28540ecd7e385178c" } new_table_name: "default.Table_2e64b9ef79734139945c53d44ce279e3" modify_external_catalogs: true
I20250905 19:52:16.741819 1184 ts_tablet_manager.cc:1939] T 6f4ec474911941b2a2d0825cdb37bd72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.724560 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet afb404e5b06c452bb98abeea95070a5f (table default.Table_3c27b9d996a542a1bed7229a301e6a9e [id=b1e354dd48554016baae491637511fa7]) successfully deleted
I20250905 19:52:16.742766 1184 tablet_replica.cc:331] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.742975 1184 raft_consensus.cc:2241] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.743130 1184 raft_consensus.cc:2270] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.743471 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6f4ec474911941b2a2d0825cdb37bd72 (table default.Table_7dd83ce427b9438990eee889da0bcb0f [id=8904fcfbf1a94032b62d596e1ea14569]) successfully deleted
I20250905 19:52:16.743669 1184 ts_tablet_manager.cc:1905] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.748528 1169 tablet_bootstrap.cc:492] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.748705 1169 ts_tablet_manager.cc:1397] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250905 19:52:16.748771 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 7e62dbe9e48544109446323b2cc4501f (table default.Table_3863927d1ffa41289c4ccd14dcec05f9 [id=567a4196a6af4e2b828347357c44313c]) successfully deleted
I20250905 19:52:16.749351 1186 tablet_replica.cc:331] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.749550 1186 raft_consensus.cc:2241] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
W20250905 19:52:16.749549 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet cbf0b20630b44230b17a80cdb53322fa because the tablet was not found. No further retry: Not found: Tablet not found: cbf0b20630b44230b17a80cdb53322fa
I20250905 19:52:16.749600 1169 raft_consensus.cc:357] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.749809 1169 raft_consensus.cc:383] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.749845 1186 pending_rounds.cc:70] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
I20250905 19:52:16.749877 1169 raft_consensus.cc:738] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.749924 1186 pending_rounds.cc:77] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089530028007424 op_type: NO_OP noop_request { }
I20250905 19:52:16.750059 1186 raft_consensus.cc:2887] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:16.750145 1169 consensus_queue.cc:260] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.750708 1186 raft_consensus.cc:2270] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.751159 1169 ts_tablet_manager.cc:1428] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:16.751456 1169 tablet_bootstrap.cc:492] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.751575 1186 ts_tablet_manager.cc:1905] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.754079 1169 tablet_bootstrap.cc:654] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.757375 1184 ts_tablet_manager.cc:1918] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.757807 1184 log.cc:1199] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/78ac5c7e9b6741d79db2e606d55a3509
I20250905 19:52:16.758255 1184 ts_tablet_manager.cc:1939] T 78ac5c7e9b6741d79db2e606d55a3509 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.758630 1169 tablet_bootstrap.cc:492] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.758776 1169 ts_tablet_manager.cc:1397] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.000s
I20250905 19:52:16.758827 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 0480eda7e20c4336bbeacad1e7cdd56a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.760218 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 78ac5c7e9b6741d79db2e606d55a3509 (table default.Table_755dce6406b647a1b2fffb9aa254e403 [id=5ac15645cb4d4730a4b4464a85299bed]) successfully deleted
I20250905 19:52:16.760416 745 tablet_service.cc:1515] Processing DeleteTablet for tablet eacedd2e38d64d2cb25b2ebd88f563a1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.761513 1169 raft_consensus.cc:357] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.761703 1169 raft_consensus.cc:383] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.761785 1169 raft_consensus.cc:738] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.761994 1169 consensus_queue.cc:260] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.762923 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ec810e5cc1af4b9d9f6bd421072bbc77
I20250905 19:52:16.763466 449 catalog_manager.cc:6652] ReplaceTablet: tablet ec810e5cc1af4b9d9f6bd421072bbc77 deleted and replaced by tablet 9477afb5b47c46d4b24128a6c4bdabc6
I20250905 19:52:16.763712 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8adc74a357244102801b2aa8d27d9747 with delete_type TABLET_DATA_DELETED (replaced by tablet 2201e4bad99643a9b041f6f99406433b) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.764269 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ec810e5cc1af4b9d9f6bd421072bbc77 with delete_type TABLET_DATA_DELETED (replaced by tablet 9477afb5b47c46d4b24128a6c4bdabc6) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.764412 1169 ts_tablet_manager.cc:1428] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:16.764621 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ec810e5cc1af4b9d9f6bd421072bbc77 with delete_type TABLET_DATA_DELETED (replaced by tablet 9477afb5b47c46d4b24128a6c4bdabc6) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.764858 1185 ts_tablet_manager.cc:1918] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.764876 1169 tablet_bootstrap.cc:492] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.765074 1185 log.cc:1199] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6f4ec474911941b2a2d0825cdb37bd72
I20250905 19:52:16.765508 1185 ts_tablet_manager.cc:1939] T 6f4ec474911941b2a2d0825cdb37bd72 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.766611 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6f4ec474911941b2a2d0825cdb37bd72 (table default.Table_7dd83ce427b9438990eee889da0bcb0f [id=8904fcfbf1a94032b62d596e1ea14569]) successfully deleted
W20250905 19:52:16.766700 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 77cbdda0e2d641bd8092f00925bb9a9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77cbdda0e2d641bd8092f00925bb9a9c already in progress: creating tablet
I20250905 19:52:16.767022 1184 tablet_replica.cc:331] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.767211 1185 tablet_replica.cc:331] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.767262 1184 raft_consensus.cc:2241] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.767467 1185 raft_consensus.cc:2241] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.767509 1184 raft_consensus.cc:2270] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.767696 1185 raft_consensus.cc:2270] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.768563 1185 ts_tablet_manager.cc:1905] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.768725 1184 ts_tablet_manager.cc:1905] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.769325 1171 tablet_bootstrap.cc:492] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.769198 1169 tablet_bootstrap.cc:654] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.769753 1171 ts_tablet_manager.cc:1397] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.082s user 0.004s sys 0.000s
I20250905 19:52:16.770606 1171 raft_consensus.cc:357] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.770820 1171 raft_consensus.cc:383] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.770941 1171 raft_consensus.cc:738] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.775278 1186 ts_tablet_manager.cc:1918] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:16.775636 1186 log.cc:1199] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8240e51124a74b81a0bdabb3ef35e66f
I20250905 19:52:16.776239 1186 ts_tablet_manager.cc:1939] T 8240e51124a74b81a0bdabb3ef35e66f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:16.776440 1171 consensus_queue.cc:260] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.777043 1171 ts_tablet_manager.cc:1428] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:16.777299 1184 ts_tablet_manager.cc:1918] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.777374 1171 tablet_bootstrap.cc:492] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.777400 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a77e895ae1804560879140ca64db74f6" } modify_external_catalogs: true
I20250905 19:52:16.777575 1184 log.cc:1199] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:16.777570 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a77e895ae1804560879140ca64db74f6" } modify_external_catalogs: true
I20250905 19:52:16.777047 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6f6a2b47de6b47f0a5eeb3d53a2addd2" } new_table_name: "default.Table_72aac5d5ddba4d23a99e5940bad14bb3" modify_external_catalogs: true
I20250905 19:52:16.781255 1184 ts_tablet_manager.cc:1939] T 57259d8eb0d44d1cbac80047c7ad67b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.782351 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 57259d8eb0d44d1cbac80047c7ad67b6 (table default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]) successfully deleted
I20250905 19:52:16.783143 1184 tablet_replica.cc:331] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.783501 1184 raft_consensus.cc:2241] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.784148 1184 raft_consensus.cc:2270] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.784940 1184 ts_tablet_manager.cc:1905] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.785591 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a5a47a603b674a41b47e7bf4b55b6a1b" } modify_external_catalogs: true
I20250905 19:52:16.785810 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a5a47a603b674a41b47e7bf4b55b6a1b" } modify_external_catalogs: true
I20250905 19:52:16.788339 1273 raft_consensus.cc:491] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.788512 1273 raft_consensus.cc:513] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.789256 1273 leader_election.cc:290] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.789978 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbab0209a48d4ad8b2f2dec0562e9598" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:16.790194 1053 raft_consensus.cc:2466] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:16.790269 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9477afb5b47c46d4b24128a6c4bdabc6 has no consensus state
I20250905 19:52:16.790683 1169 tablet_bootstrap.cc:492] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.790688 849 leader_election.cc:304] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.790851 1169 ts_tablet_manager.cc:1397] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.000s
W20250905 19:52:16.791182 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fac0e1f483b40ab8d66cb5a9ee808fc consensus state has no leader
I20250905 19:52:16.791584 1273 raft_consensus.cc:2802] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:16.791709 1273 raft_consensus.cc:491] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:16.791680 1169 raft_consensus.cc:357] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.791814 1273 raft_consensus.cc:3058] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.791885 1169 raft_consensus.cc:383] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.791996 1169 raft_consensus.cc:738] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.792250 1169 consensus_queue.cc:260] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.792268 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_59292ef8849c475b9de70c4c9480daaf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.792764 1169 ts_tablet_manager.cc:1428] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250905 19:52:16.792843 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_59292ef8849c475b9de70c4c9480daaf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.795141 1273 raft_consensus.cc:513] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.796422 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a39be9af778c4fa49c2376cb4bd7397b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.796478 1273 leader_election.cc:290] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:16.797133 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a39be9af778c4fa49c2376cb4bd7397b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.798640 1171 tablet_bootstrap.cc:654] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.799511 1169 tablet_bootstrap.cc:492] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.799546 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbab0209a48d4ad8b2f2dec0562e9598" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:16.799754 1053 raft_consensus.cc:3058] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:16.801525 1169 tablet_bootstrap.cc:654] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:16.802673 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2466a1bafc8347e483ca5e28d552bfc9 has no consensus state
I20250905 19:52:16.802693 1053 raft_consensus.cc:2466] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:16.803313 1176 raft_consensus.cc:491] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.803468 1176 raft_consensus.cc:513] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.803740 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_cddec42a8c1c415ba27927b52d055a08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.804183 1176 leader_election.cc:290] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
W20250905 19:52:16.804348 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cddec42a8c1c415ba27927b52d055a08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.804766 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:16.805167 849 leader_election.cc:304] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:16.805899 1184 ts_tablet_manager.cc:1918] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.806041 1184 log.cc:1199] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ec810e5cc1af4b9d9f6bd421072bbc77
W20250905 19:52:16.806226 986 leader_election.cc:343] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.806421 1184 ts_tablet_manager.cc:1939] T ec810e5cc1af4b9d9f6bd421072bbc77 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.807720 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:16.808564 1273 raft_consensus.cc:2802] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:16.808853 1273 raft_consensus.cc:695] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:16.810966 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b93afdd4ab954f588fabeb7cfe5713a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.811570 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b93afdd4ab954f588fabeb7cfe5713a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.815083 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c0604f9fc0a94c19b6079eadec32b3ad"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.815596 1273 consensus_queue.cc:237] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.821420 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a2268b9408854caaac5ebd773623563b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.822141 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a2268b9408854caaac5ebd773623563b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.824632 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ae188650e4914c02837028a3645a3b48" } modify_external_catalogs: true
I20250905 19:52:16.824858 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ae188650e4914c02837028a3645a3b48" } modify_external_catalogs: true
W20250905 19:52:16.826506 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c0604f9fc0a94c19b6079eadec32b3ad in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.831672 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 757a7d3fba8d43dda36474539b7296ba (DEFAULT_TABLE table=default.Table_a5a47a603b674a41b47e7bf4b55b6a1b [id=106f635aee6b4cd4a7f7b1af9088a286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.831818 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_10b6351477504574bbcdef1d7f59c24a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.832178 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757a7d3fba8d43dda36474539b7296ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.779721 745 tablet_service.cc:1515] Processing DeleteTablet for tablet ec810e5cc1af4b9d9f6bd421072bbc77 with delete_type TABLET_DATA_DELETED (replaced by tablet 9477afb5b47c46d4b24128a6c4bdabc6) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:16.832548 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10b6351477504574bbcdef1d7f59c24a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.789791 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbab0209a48d4ad8b2f2dec0562e9598" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.796296 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbab0209a48d4ad8b2f2dec0562e9598" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:16.806257 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.834132 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8240e51124a74b81a0bdabb3ef35e66f (table default.Table_d1558a276407476f80051a63eea7bdaf [id=ff84de6d110a451c9b8da5ce3803b1f2]) successfully deleted
I20250905 19:52:16.835440 745 tablet_service.cc:1468] Processing CreateTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 (DEFAULT_TABLE table=default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.835855 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a32f742eaf473e9f6f9bf214f53679. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:16.836860 984 leader_election.cc:343] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.837000 984 leader_election.cc:304] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.837111 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b (DEFAULT_TABLE table=default.Table_e73ba1d781524aa7a6c1714e3af86d7d [id=ff5964a447094f6ea231fb91971d83e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.837620 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43111a3512fb4fc0bdf4ee245b16695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.837957 1285 raft_consensus.cc:491] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.838122 1285 raft_consensus.cc:513] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.838848 1285 leader_election.cc:290] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:16.839255 893 tablet_service.cc:1468] Processing CreateTablet for tablet 757a7d3fba8d43dda36474539b7296ba (DEFAULT_TABLE table=default.Table_a5a47a603b674a41b47e7bf4b55b6a1b [id=106f635aee6b4cd4a7f7b1af9088a286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.839630 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757a7d3fba8d43dda36474539b7296ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.840257 888 tablet_service.cc:1468] Processing CreateTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b (DEFAULT_TABLE table=default.Table_e73ba1d781524aa7a6c1714e3af86d7d [id=ff5964a447094f6ea231fb91971d83e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.840606 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43111a3512fb4fc0bdf4ee245b16695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.843025 1236 raft_consensus.cc:2747] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.844870 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8adc74a357244102801b2aa8d27d9747" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:16.845402 848 leader_election.cc:343] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:16.845760 848 leader_election.cc:343] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.846024 891 tablet_service.cc:1468] Processing CreateTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 (DEFAULT_TABLE table=default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:16.846268 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 57259d8eb0d44d1cbac80047c7ad67b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57259d8eb0d44d1cbac80047c7ad67b6 already in progress: creating tablet
I20250905 19:52:16.846385 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a32f742eaf473e9f6f9bf214f53679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.848073 1186 tablet_replica.cc:331] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.848202 1032 tablet_service.cc:1468] Processing CreateTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 (DEFAULT_TABLE table=default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.848273 1186 raft_consensus.cc:2241] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:16.848265 849 leader_election.cc:343] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 8adc74a357244102801b2aa8d27d9747
I20250905 19:52:16.848460 1186 raft_consensus.cc:2270] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:16.848522 848 leader_election.cc:343] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.848618 848 leader_election.cc:304] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.848578 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8a32f742eaf473e9f6f9bf214f53679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.849274 1285 raft_consensus.cc:2747] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.853006 745 tablet_service.cc:1468] Processing CreateTablet for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (DEFAULT_TABLE table=default.Table_702bc880676a4ad3b61ef0ec4051a032 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.853596 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5748bea3bd4f51a0d6c28a1d9044a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.860306 888 tablet_service.cc:1468] Processing CreateTablet for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (DEFAULT_TABLE table=default.Table_702bc880676a4ad3b61ef0ec4051a032 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.860867 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5748bea3bd4f51a0d6c28a1d9044a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.861954 755 tablet_service.cc:1468] Processing CreateTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b (DEFAULT_TABLE table=default.Table_e73ba1d781524aa7a6c1714e3af86d7d [id=ff5964a447094f6ea231fb91971d83e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.862432 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43111a3512fb4fc0bdf4ee245b16695b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.867884 754 tablet_service.cc:1468] Processing CreateTablet for tablet 757a7d3fba8d43dda36474539b7296ba (DEFAULT_TABLE table=default.Table_a5a47a603b674a41b47e7bf4b55b6a1b [id=106f635aee6b4cd4a7f7b1af9088a286]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.867996 1026 tablet_service.cc:1468] Processing CreateTablet for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (DEFAULT_TABLE table=default.Table_702bc880676a4ad3b61ef0ec4051a032 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.868455 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 757a7d3fba8d43dda36474539b7296ba. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.868602 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af5748bea3bd4f51a0d6c28a1d9044a0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.869541 746 tablet_service.cc:1468] Processing CreateTablet for tablet 087172093fff4c41980dc2daeabb07c3 (DEFAULT_TABLE table=default.Table_fd280a2a9f5f4f81ad7dcd3fd2b4003b [id=d7c0f44965694f77a8eae2ec4dd18a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.870029 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 087172093fff4c41980dc2daeabb07c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.875121 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 087172093fff4c41980dc2daeabb07c3 (DEFAULT_TABLE table=default.Table_fd280a2a9f5f4f81ad7dcd3fd2b4003b [id=d7c0f44965694f77a8eae2ec4dd18a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.875566 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 087172093fff4c41980dc2daeabb07c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.875663 753 tablet_service.cc:1468] Processing CreateTablet for tablet d480531ba94f4b0f9fe40e88fe91512e (DEFAULT_TABLE table=default.Table_c7b1f5de2f2f4657971e14388d57ef5a [id=d92e1dc4017f4f8cbfdeeb40900f136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.876062 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d480531ba94f4b0f9fe40e88fe91512e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.876880 452 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 686de0758fdc45589009924322efc1b7
I20250905 19:52:16.877303 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b86b0c6f5fe4496ead402827d2db3f1d" } modify_external_catalogs: true
I20250905 19:52:16.877478 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b86b0c6f5fe4496ead402827d2db3f1d" } modify_external_catalogs: true
I20250905 19:52:16.881678 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 6d12c7fbb84849c68274810b3fdcba04 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:16.885021 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d23d6df63c1a4e02a43bccf625e200bb
I20250905 19:52:16.885560 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:16.885854 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_6eb260fb20e14dc59c7f4bdb683fa538 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 1 got 0
I20250905 19:52:16.886063 744 tablet_service.cc:1515] Processing DeleteTablet for tablet d23d6df63c1a4e02a43bccf625e200bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.886142 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]). Expected version 1 got 0
I20250905 19:52:16.886399 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]). Expected version 1 got 0
I20250905 19:52:16.886637 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:16.886927 447 catalog_manager.cc:5582] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:16.887758 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 757a7d3fba8d43dda36474539b7296ba
W20250905 19:52:16.888878 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 09be30768f5742cf9a4bf1e3fe86dfe8 consensus state has no leader
I20250905 19:52:16.889158 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.889187 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:16.889601 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 85c648f799304fa9a94bec665a2dcf97 consensus state has no leader
I20250905 19:52:16.889895 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:16.890368 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 757a7d3fba8d43dda36474539b7296ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 757a7d3fba8d43dda36474539b7296ba already in progress: creating tablet
I20250905 19:52:16.890610 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet d23d6df63c1a4e02a43bccf625e200bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.890970 1184 tablet_replica.cc:331] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:16.891183 1184 raft_consensus.cc:2241] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.891368 1184 raft_consensus.cc:2270] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.891945 1184 ts_tablet_manager.cc:1905] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.892035 891 tablet_service.cc:1468] Processing CreateTablet for tablet d480531ba94f4b0f9fe40e88fe91512e (DEFAULT_TABLE table=default.Table_c7b1f5de2f2f4657971e14388d57ef5a [id=d92e1dc4017f4f8cbfdeeb40900f136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.892463 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d480531ba94f4b0f9fe40e88fe91512e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.894236 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a3ae86cb11e24d348e2f12e982f3ab9f
I20250905 19:52:16.894714 446 catalog_manager.cc:6652] ReplaceTablet: tablet a3ae86cb11e24d348e2f12e982f3ab9f deleted and replaced by tablet 544dffc8b77049c295921d0af091b17d
I20250905 19:52:16.894938 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f with delete_type TABLET_DATA_DELETED (replaced by tablet 544dffc8b77049c295921d0af091b17d) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.897305 892 tablet_service.cc:1515] Processing DeleteTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f with delete_type TABLET_DATA_DELETED (replaced by tablet 544dffc8b77049c295921d0af091b17d) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.899507 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_07f86749b59347fbb1060d7102299913" } modify_external_catalogs: true
I20250905 19:52:16.899734 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_07f86749b59347fbb1060d7102299913" } modify_external_catalogs: true
I20250905 19:52:16.899895 1176 raft_consensus.cc:491] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.900117 1176 raft_consensus.cc:513] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.900765 1176 leader_election.cc:290] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:16.901003 1184 ts_tablet_manager.cc:1918] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.901186 1184 log.cc:1199] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d23d6df63c1a4e02a43bccf625e200bb
I20250905 19:52:16.901561 1184 ts_tablet_manager.cc:1939] T d23d6df63c1a4e02a43bccf625e200bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:16.901657 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be73873da994402e84fff6aa2ded7525" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:16.902297 986 leader_election.cc:343] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.902557 1185 ts_tablet_manager.cc:1918] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.902695 1185 log.cc:1199] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/b776063b81c44217b517a619af927c08
I20250905 19:52:16.902662 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be73873da994402e84fff6aa2ded7525" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.902961 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d23d6df63c1a4e02a43bccf625e200bb (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]) successfully deleted
I20250905 19:52:16.903059 1185 ts_tablet_manager.cc:1939] T b776063b81c44217b517a619af927c08 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:16.903235 984 leader_election.cc:343] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.903337 984 leader_election.cc:304] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.903865 1176 raft_consensus.cc:2747] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:16.904054 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet b776063b81c44217b517a619af927c08 (table default.Table_ce8e455023774d02ba896de13623c0ea [id=06e2f51841544340adc3a39ca41a9bea]) successfully deleted
I20250905 19:52:16.904109 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f with delete_type TABLET_DATA_DELETED (replaced by tablet 544dffc8b77049c295921d0af091b17d) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.904305 1185 tablet_replica.cc:331] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.904541 1185 raft_consensus.cc:2241] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:16.904546 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 77cbdda0e2d641bd8092f00925bb9a9c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 77cbdda0e2d641bd8092f00925bb9a9c already in progress: creating tablet
W20250905 19:52:16.904789 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 57259d8eb0d44d1cbac80047c7ad67b6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57259d8eb0d44d1cbac80047c7ad67b6 already in progress: creating tablet
I20250905 19:52:16.904814 1185 raft_consensus.cc:2270] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.892455 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d23d6df63c1a4e02a43bccf625e200bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.905536 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 7edcf57f7753449d8a5fcdb6f4f1dc21
W20250905 19:52:16.905637 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a3ae86cb11e24d348e2f12e982f3ab9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ae86cb11e24d348e2f12e982f3ab9f already in progress: creating tablet
I20250905 19:52:16.906473 1026 tablet_service.cc:1468] Processing CreateTablet for tablet d480531ba94f4b0f9fe40e88fe91512e (DEFAULT_TABLE table=default.Table_c7b1f5de2f2f4657971e14388d57ef5a [id=d92e1dc4017f4f8cbfdeeb40900f136c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:16.906874 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d480531ba94f4b0f9fe40e88fe91512e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:16.907071 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.907892 1185 ts_tablet_manager.cc:1905] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.909842 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:16.909996 1186 ts_tablet_manager.cc:1905] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.910588 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.911135 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:16.912606 450 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet afa17a769daa4266995a4335d8468525
W20250905 19:52:16.913244 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 already in progress: creating tablet
I20250905 19:52:16.913524 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.914497 1185 ts_tablet_manager.cc:1918] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.914620 1185 log.cc:1199] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8adc74a357244102801b2aa8d27d9747
I20250905 19:52:16.914988 1185 ts_tablet_manager.cc:1939] T 8adc74a357244102801b2aa8d27d9747 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.916162 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8adc74a357244102801b2aa8d27d9747 (table default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]) successfully deleted
I20250905 19:52:16.916798 454 catalog_manager.cc:5582] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:16.917275 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]). Expected version 1 got 0
I20250905 19:52:16.917393 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:16.917455 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:16.916797 1185 tablet_replica.cc:331] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.917936 1185 raft_consensus.cc:2241] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.918231 1185 raft_consensus.cc:2270] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.918836 1185 ts_tablet_manager.cc:1905] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.924428 1236 raft_consensus.cc:491] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:16.924593 1236 raft_consensus.cc:513] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.925264 1236 leader_election.cc:290] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:16.925884 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8d3cbdda0c94e139259d90919dc2f95" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:16.926496 986 leader_election.cc:343] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.927273 1171 tablet_bootstrap.cc:492] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.927227 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8d3cbdda0c94e139259d90919dc2f95" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:16.927238 1185 ts_tablet_manager.cc:1918] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.927599 1185 log.cc:1199] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ec810e5cc1af4b9d9f6bd421072bbc77
I20250905 19:52:16.928030 1185 ts_tablet_manager.cc:1939] T ec810e5cc1af4b9d9f6bd421072bbc77 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:16.928115 984 leader_election.cc:343] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:16.928278 984 leader_election.cc:304] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:16.929164 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:16.927436 1171 ts_tablet_manager.cc:1397] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.150s user 0.000s sys 0.005s
I20250905 19:52:16.930262 1236 raft_consensus.cc:2747] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:16.930408 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 757a7d3fba8d43dda36474539b7296ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 757a7d3fba8d43dda36474539b7296ba already in progress: creating tablet
I20250905 19:52:16.930589 1170 tablet_bootstrap.cc:492] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:16.930780 1170 ts_tablet_manager.cc:1397] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.271s user 0.008s sys 0.001s
I20250905 19:52:16.931597 1170 raft_consensus.cc:357] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.931890 1170 raft_consensus.cc:383] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.932063 1170 raft_consensus.cc:738] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.933154 1185 tablet_replica.cc:331] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:16.933827 1185 raft_consensus.cc:2241] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.934103 1185 raft_consensus.cc:2270] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:16.933559 1171 raft_consensus.cc:357] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.934536 1171 raft_consensus.cc:383] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.934639 1170 consensus_queue.cc:260] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.935328 1171 raft_consensus.cc:738] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.935576 1171 consensus_queue.cc:260] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.935235 1185 ts_tablet_manager.cc:1905] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.937623 1171 ts_tablet_manager.cc:1428] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20250905 19:52:16.937952 1171 tablet_bootstrap.cc:492] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.939260 1170 ts_tablet_manager.cc:1428] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 19:52:16.939416 453 catalog_manager.cc:6652] ReplaceTablet: tablet 6d12c7fbb84849c68274810b3fdcba04 deleted and replaced by tablet a5f076d9917f40d899d7422ad974fe0f
W20250905 19:52:16.939513 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a3ae86cb11e24d348e2f12e982f3ab9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ae86cb11e24d348e2f12e982f3ab9f already in progress: creating tablet
W20250905 19:52:16.939409 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6d12c7fbb84849c68274810b3fdcba04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d12c7fbb84849c68274810b3fdcba04 already in progress: creating tablet
I20250905 19:52:16.941318 1169 tablet_bootstrap.cc:492] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:16.941468 1169 ts_tablet_manager.cc:1397] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.148s user 0.021s sys 0.048s
I20250905 19:52:16.942063 1171 tablet_bootstrap.cc:654] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.942342 1169 raft_consensus.cc:357] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.942662 1169 raft_consensus.cc:383] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.942764 1169 raft_consensus.cc:738] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.942973 1169 consensus_queue.cc:260] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.944748 1169 ts_tablet_manager.cc:1428] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:16.945354 1169 tablet_bootstrap.cc:492] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:16.948158 1185 ts_tablet_manager.cc:1918] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.948544 1185 log.cc:1199] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d23d6df63c1a4e02a43bccf625e200bb
I20250905 19:52:16.949075 1185 ts_tablet_manager.cc:1939] T d23d6df63c1a4e02a43bccf625e200bb P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:16.950893 1170 tablet_bootstrap.cc:492] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:16.954566 1171 tablet_bootstrap.cc:492] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.954713 1171 ts_tablet_manager.cc:1397] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250905 19:52:16.955479 1171 raft_consensus.cc:357] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.955664 1171 raft_consensus.cc:383] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.955757 1171 raft_consensus.cc:738] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.956032 1171 consensus_queue.cc:260] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.959491 1171 ts_tablet_manager.cc:1428] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:16.960646 1169 tablet_bootstrap.cc:654] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.962234 1171 tablet_bootstrap.cc:492] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.959096 1186 ts_tablet_manager.cc:1918] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:16.963244 424 master_runner.cc:186] Error getting master registration for 127.0.104.60:34395: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111),
I20250905 19:52:16.963394 1186 log.cc:1199] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/0480eda7e20c4336bbeacad1e7cdd56a
I20250905 19:52:16.963447 424 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20250905 19:52:16.963555 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d23d6df63c1a4e02a43bccf625e200bb (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]) successfully deleted
W20250905 19:52:16.963845 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 already in progress: creating tablet
I20250905 19:52:16.964072 1186 ts_tablet_manager.cc:1939] T 0480eda7e20c4336bbeacad1e7cdd56a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:16.964339 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6d12c7fbb84849c68274810b3fdcba04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d12c7fbb84849c68274810b3fdcba04 already in progress: creating tablet
I20250905 19:52:16.964612 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 2 got 0
I20250905 19:52:16.964849 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 1c6295c8c7874df89662b8cef6459fec (table default.Table_0de80dfdff2546a2a39368585f13a394 [id=2151f6f3348c4d1fb376448ecfcbae8a]). Expected version 1 got 0
I20250905 19:52:16.965158 1171 tablet_bootstrap.cc:654] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.970520 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 1c6295c8c7874df89662b8cef6459fec (table default.Table_0de80dfdff2546a2a39368585f13a394 [id=2151f6f3348c4d1fb376448ecfcbae8a]). Expected version 1 got 0
I20250905 19:52:16.971267 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 0480eda7e20c4336bbeacad1e7cdd56a (table default.Table_6c729e17dd76402eb3cd6ed85f3d9875 [id=d9c9bed59ab9499393be43c41657e6e0]) successfully deleted
W20250905 19:52:16.974092 712 consensus_peers.cc:489] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ec810e5cc1af4b9d9f6bd421072bbc77. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:16.974293 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a659dd4a6bcf4f618d1ddec59f33ffca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.974464 1186 tablet_replica.cc:331] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:16.974712 1186 raft_consensus.cc:2241] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:16.974663 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9f1863dcd87d4fb98b0f2e89677b5edf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:16.974913 1186 raft_consensus.cc:2270] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:16.974920 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a659dd4a6bcf4f618d1ddec59f33ffca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.975036 1171 tablet_bootstrap.cc:492] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:16.975095 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9f1863dcd87d4fb98b0f2e89677b5edf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.975198 1171 ts_tablet_manager.cc:1397] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.007s
I20250905 19:52:16.976004 1171 raft_consensus.cc:357] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.976219 1171 raft_consensus.cc:383] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:16.976302 1171 raft_consensus.cc:738] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:16.974293 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7240f6ab8dbf4c16a2651dfc5a92d15b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:16.977223 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7240f6ab8dbf4c16a2651dfc5a92d15b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:16.976553 1171 consensus_queue.cc:260] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:16.978834 1186 ts_tablet_manager.cc:1905] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:16.982961 1171 ts_tablet_manager.cc:1428] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250905 19:52:16.986053 1170 tablet_bootstrap.cc:654] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.988186 1171 tablet_bootstrap.cc:492] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:16.990067 1171 tablet_bootstrap.cc:654] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:16.992933 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_702bc880676a4ad3b61ef0ec4051a032" } new_table_name: "default.Table_33489afbbd2a47cfa0eaea7cfcba8f17" modify_external_catalogs: true
I20250905 19:52:16.995994 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e73ba1d781524aa7a6c1714e3af86d7d" } new_table_name: "default.Table_d7e9ca2342f94ec49cb959b60816c1d3" modify_external_catalogs: true
I20250905 19:52:16.996420 1362 tablet.cc:1722] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:16.998833 1171 tablet_bootstrap.cc:492] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:16.999075 1171 ts_tablet_manager.cc:1397] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.003s
I20250905 19:52:16.999987 1171 raft_consensus.cc:357] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.000175 1171 raft_consensus.cc:383] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.000278 1171 raft_consensus.cc:738] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.000540 1171 consensus_queue.cc:260] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.000666 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af5748bea3bd4f51a0d6c28a1d9044a0 consensus state has no leader
I20250905 19:52:17.000619 1361 tablet.cc:1722] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:17.005272 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_59292ef8849c475b9de70c4c9480daaf" } new_table_name: "default.Table_929cfcb2535e452ebb48483e593faa7a" modify_external_catalogs: true
I20250905 19:52:17.005746 1171 ts_tablet_manager.cc:1428] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
W20250905 19:52:17.006918 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 43111a3512fb4fc0bdf4ee245b16695b consensus state has no leader
I20250905 19:52:17.007058 888 tablet_service.cc:1468] Processing CreateTablet for tablet 087172093fff4c41980dc2daeabb07c3 (DEFAULT_TABLE table=default.Table_fd280a2a9f5f4f81ad7dcd3fd2b4003b [id=d7c0f44965694f77a8eae2ec4dd18a03]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.007536 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 087172093fff4c41980dc2daeabb07c3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.008152 1169 tablet_bootstrap.cc:492] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.008611 1171 tablet_bootstrap.cc:492] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:17.008332 1169 ts_tablet_manager.cc:1397] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.063s user 0.004s sys 0.004s
I20250905 19:52:17.009557 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6a7e530e3936440fa898546212620a3b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.010133 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a7e530e3936440fa898546212620a3b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.010843 1171 tablet_bootstrap.cc:654] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.012307 1203 raft_consensus.cc:491] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.012558 1203 raft_consensus.cc:513] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.013126 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 2f581403ec924fb0948400df41ed8aab has no consensus state
I20250905 19:52:17.013353 1203 leader_election.cc:290] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:17.014168 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2c7906a718c46d9a76587a70d96a1f9" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:17.016664 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b2c7906a718c46d9a76587a70d96a1f9" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.017153 913 raft_consensus.cc:2391] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:17.017792 712 leader_election.cc:304] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:17.018473 1203 raft_consensus.cc:3058] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.021770 1169 raft_consensus.cc:357] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.022583 1203 raft_consensus.cc:2747] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.022742 1169 raft_consensus.cc:383] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.022855 1169 raft_consensus.cc:738] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.023212 1169 consensus_queue.cc:260] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.024173 1169 ts_tablet_manager.cc:1428] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250905 19:52:17.024627 1169 tablet_bootstrap.cc:492] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.028631 1169 tablet_bootstrap.cc:654] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.033466 1171 tablet_bootstrap.cc:492] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:17.033699 1171 ts_tablet_manager.cc:1397] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.007s
I20250905 19:52:17.036109 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_58b0d391a4044d39951ca7f9799bcf70"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.036821 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58b0d391a4044d39951ca7f9799bcf70 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.036890 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b96ef31f0115467fb13cceb26c34bc99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.037382 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b96ef31f0115467fb13cceb26c34bc99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.038208 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_5918e71853e2403a9d36c683ed58ac88"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:17.038511 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6fc6ab8b63df4d2684a474b096058aee"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.039251 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5918e71853e2403a9d36c683ed58ac88 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:17.039247 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6fc6ab8b63df4d2684a474b096058aee in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.041486 1171 raft_consensus.cc:357] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.041815 1171 raft_consensus.cc:383] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.041965 1171 raft_consensus.cc:738] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.042131 1236 consensus_queue.cc:1035] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:17.042297 1171 consensus_queue.cc:260] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.045763 1171 ts_tablet_manager.cc:1428] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.012s user 0.002s sys 0.001s
I20250905 19:52:17.048695 1171 tablet_bootstrap.cc:492] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:17.050676 1170 tablet_bootstrap.cc:492] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:17.050879 1170 ts_tablet_manager.cc:1397] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.100s user 0.002s sys 0.002s
I20250905 19:52:17.051605 1171 tablet_bootstrap.cc:654] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.051666 1170 raft_consensus.cc:357] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.051854 1170 raft_consensus.cc:383] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.051918 1170 raft_consensus.cc:738] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.052199 1170 consensus_queue.cc:260] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.052340 1169 tablet_bootstrap.cc:492] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.052492 1169 ts_tablet_manager.cc:1397] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.007s
I20250905 19:52:17.052691 1170 ts_tablet_manager.cc:1428] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:17.053705 1169 raft_consensus.cc:357] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.053961 1169 raft_consensus.cc:383] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.054069 1169 raft_consensus.cc:738] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.054132 1176 consensus_queue.cc:1035] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:17.054373 1169 consensus_queue.cc:260] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.055475 1169 ts_tablet_manager.cc:1428] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:17.055779 1169 tablet_bootstrap.cc:492] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.058670 1169 tablet_bootstrap.cc:654] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.064850 1170 tablet_bootstrap.cc:492] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:17.067289 1170 tablet_bootstrap.cc:654] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.070346 1169 tablet_bootstrap.cc:492] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.070545 1169 ts_tablet_manager.cc:1397] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.002s
I20250905 19:52:17.071513 1169 raft_consensus.cc:357] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.071794 1169 raft_consensus.cc:383] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.071872 1169 raft_consensus.cc:738] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.072166 1169 consensus_queue.cc:260] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.077239 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 266ffb862d144368994f5c33e684647d from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.084496 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 59faa6716fed4ea6b563df5f024b2058 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.090535 1169 ts_tablet_manager.cc:1428] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250905 19:52:17.091204 1169 tablet_bootstrap.cc:492] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.096622 1186 ts_tablet_manager.cc:1918] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.096787 1186 log.cc:1199] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/eacedd2e38d64d2cb25b2ebd88f563a1
I20250905 19:52:17.097913 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c7b1f5de2f2f4657971e14388d57ef5a" } modify_external_catalogs: true
I20250905 19:52:17.098057 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 6d09371944644b3082268b2b37ce54f8 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.098150 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c7b1f5de2f2f4657971e14388d57ef5a" } modify_external_catalogs: true
I20250905 19:52:17.098472 1186 ts_tablet_manager.cc:1939] T eacedd2e38d64d2cb25b2ebd88f563a1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:17.098649 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a" } modify_external_catalogs: true
I20250905 19:52:17.098799 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a" } modify_external_catalogs: true
I20250905 19:52:17.098990 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5996e867324248b1b6c68868f6dfe8e2" } modify_external_catalogs: true
I20250905 19:52:17.099128 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5996e867324248b1b6c68868f6dfe8e2" } modify_external_catalogs: true
I20250905 19:52:17.099639 1186 tablet_replica.cc:331] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:17.099819 1186 raft_consensus.cc:2241] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:17.100226 1186 pending_rounds.cc:70] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
I20250905 19:52:17.100314 1186 pending_rounds.cc:77] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089530578223104 op_type: NO_OP noop_request { }
I20250905 19:52:17.100447 1186 raft_consensus.cc:2887] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:17.100535 1186 raft_consensus.cc:2270] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.095770 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_619041804f364e1d8be97fbf6e0ed649"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.101730 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_619041804f364e1d8be97fbf6e0ed649 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.101799 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet eacedd2e38d64d2cb25b2ebd88f563a1 (table default.Table_056e9cf230e44fc5badd0c9b709c0c4f [id=88e8e8eaaefb4045b64fd8ad64f9990c]) successfully deleted
I20250905 19:52:17.106431 1169 tablet_bootstrap.cc:654] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.107640 1176 raft_consensus.cc:491] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.107800 1176 raft_consensus.cc:513] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.108628 1176 leader_election.cc:290] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.109091 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c0604f9fc0a94c19b6079eadec32b3ad" } new_table_name: "default.Table_48447ad862474825a10962298c97650d" modify_external_catalogs: true
I20250905 19:52:17.109526 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.110256 984 leader_election.cc:343] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.110636 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:17.111214 986 leader_election.cc:343] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.111392 986 leader_election.cc:304] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.111953 1176 raft_consensus.cc:2747] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.114274 1176 raft_consensus.cc:491] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.114419 1176 raft_consensus.cc:513] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.114964 1176 leader_election.cc:290] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.115489 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c648f799304fa9a94bec665a2dcf97" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.116089 984 leader_election.cc:343] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.116806 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c648f799304fa9a94bec665a2dcf97" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.117122 914 raft_consensus.cc:2466] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:17.117587 986 leader_election.cc:304] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.118110 1176 raft_consensus.cc:2802] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:17.118073 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 4f1d960461ef4829960baa5cca7f7765 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.118224 1176 raft_consensus.cc:491] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:17.118311 1176 raft_consensus.cc:3058] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.118763 1171 tablet_bootstrap.cc:492] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:17.118943 1171 ts_tablet_manager.cc:1397] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.070s user 0.022s sys 0.018s
I20250905 19:52:17.119812 1171 raft_consensus.cc:357] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.120007 1171 raft_consensus.cc:383] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.120085 1171 raft_consensus.cc:738] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.120319 1171 consensus_queue.cc:260] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.121346 1171 ts_tablet_manager.cc:1428] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:17.121657 1171 tablet_bootstrap.cc:492] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:17.123453 1171 tablet_bootstrap.cc:654] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.124104 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 266ffb862d144368994f5c33e684647d
I20250905 19:52:17.124572 1176 raft_consensus.cc:513] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.124766 448 catalog_manager.cc:6652] ReplaceTablet: tablet 266ffb862d144368994f5c33e684647d deleted and replaced by tablet 0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:17.125679 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c648f799304fa9a94bec665a2dcf97" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:17.125911 913 raft_consensus.cc:3058] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.126848 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 266ffb862d144368994f5c33e684647d with delete_type TABLET_DATA_DELETED (replaced by tablet 0267ac3e3f1a45978807737eca5d91ef) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.127208 1184 tablet_replica.cc:331] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:17.127514 1184 raft_consensus.cc:2241] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.127797 1184 raft_consensus.cc:2270] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.128556 1184 ts_tablet_manager.cc:1905] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.129359 913 raft_consensus.cc:2466] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:17.129967 986 leader_election.cc:304] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:17.130321 1236 raft_consensus.cc:2802] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:17.134210 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "85c648f799304fa9a94bec665a2dcf97" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:17.134949 984 leader_election.cc:343] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.135411 1176 leader_election.cc:290] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.136170 1186 ts_tablet_manager.cc:1905] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.137388 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 266ffb862d144368994f5c33e684647d with delete_type TABLET_DATA_DELETED (replaced by tablet 0267ac3e3f1a45978807737eca5d91ef) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.141381 1376 tablet.cc:1722] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:17.144049 1236 raft_consensus.cc:695] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:17.144379 1236 consensus_queue.cc:237] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.146265 1170 tablet_bootstrap.cc:492] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:17.155642 1170 ts_tablet_manager.cc:1397] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.091s user 0.004s sys 0.003s
I20250905 19:52:17.160583 1171 tablet_bootstrap.cc:492] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:17.160765 1171 ts_tablet_manager.cc:1397] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.007s
I20250905 19:52:17.161485 1171 raft_consensus.cc:357] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.162231 1171 raft_consensus.cc:383] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.162336 1171 raft_consensus.cc:738] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.162469 1170 raft_consensus.cc:357] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.162647 1171 consensus_queue.cc:260] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.164041 1171 ts_tablet_manager.cc:1428] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250905 19:52:17.164197 1170 raft_consensus.cc:383] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.165016 1171 tablet_bootstrap.cc:492] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:17.165756 1170 raft_consensus.cc:738] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.166394 1170 consensus_queue.cc:260] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.172488 1170 ts_tablet_manager.cc:1428] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250905 19:52:17.173736 1171 tablet_bootstrap.cc:654] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.174557 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 266ffb862d144368994f5c33e684647d with delete_type TABLET_DATA_DELETED (replaced by tablet 0267ac3e3f1a45978807737eca5d91ef) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.175254 1185 tablet_replica.cc:331] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:17.175506 1185 raft_consensus.cc:2241] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.175734 1185 raft_consensus.cc:2270] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.176365 1185 ts_tablet_manager.cc:1905] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.177107 1170 tablet_bootstrap.cc:492] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:17.177697 1184 ts_tablet_manager.cc:1918] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:17.178220 1184 log.cc:1199] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/266ffb862d144368994f5c33e684647d
I20250905 19:52:17.178848 1184 ts_tablet_manager.cc:1939] T 266ffb862d144368994f5c33e684647d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:17.180481 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]) successfully deleted
I20250905 19:52:17.182474 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_46df29d8bee2462fb784be95829c6ad0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.183085 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_46df29d8bee2462fb784be95829c6ad0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.183138 1170 tablet_bootstrap.cc:654] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.183444 1185 ts_tablet_manager.cc:1918] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:17.183612 1185 log.cc:1199] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/266ffb862d144368994f5c33e684647d
I20250905 19:52:17.184072 1185 ts_tablet_manager.cc:1939] T 266ffb862d144368994f5c33e684647d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:17.184769 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a39be9af778c4fa49c2376cb4bd7397b" } modify_external_catalogs: true
I20250905 19:52:17.185191 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]) successfully deleted
I20250905 19:52:17.185497 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a39be9af778c4fa49c2376cb4bd7397b" } modify_external_catalogs: true
I20250905 19:52:17.186477 1169 tablet_bootstrap.cc:492] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.186684 1169 ts_tablet_manager.cc:1397] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.096s user 0.018s sys 0.051s
I20250905 19:52:17.186890 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_755dce6406b647a1b2fffb9aa254e403" } new_table_name: "default.Table_d6e9e58d90324cd2a7110d05c7588160" modify_external_catalogs: true
I20250905 19:52:17.187438 1169 raft_consensus.cc:357] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.187590 1169 raft_consensus.cc:383] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.187683 1169 raft_consensus.cc:738] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.187979 1169 consensus_queue.cc:260] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.190497 1169 ts_tablet_manager.cc:1428] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250905 19:52:17.190939 1169 tablet_bootstrap.cc:492] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.195397 1169 tablet_bootstrap.cc:654] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.196014 453 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]). Expected version 1 got 0
W20250905 19:52:17.196846 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4c76077e4adf45a7adc736b0e69032eb consensus state has no leader
I20250905 19:52:17.196864 1186 ts_tablet_manager.cc:1918] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:17.197016 1186 log.cc:1199] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ec810e5cc1af4b9d9f6bd421072bbc77
I20250905 19:52:17.197171 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1fd86ae5263542ed92214968ec16ccf0" } modify_external_catalogs: true
I20250905 19:52:17.197408 1186 ts_tablet_manager.cc:1939] T ec810e5cc1af4b9d9f6bd421072bbc77 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:17.197358 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1fd86ae5263542ed92214968ec16ccf0" } modify_external_catalogs: true
I20250905 19:52:17.198056 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 59faa6716fed4ea6b563df5f024b2058 (table default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 1 got 0
I20250905 19:52:17.198431 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]). Expected version 1 got 0
I20250905 19:52:17.198642 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 09b46a12076e492e89d0c0178bcf0524 (table default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a [id=3b8967eb30ad46cdbf272136f23318f1]). Expected version 1 got 0
I20250905 19:52:17.200770 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f (DEFAULT_TABLE table=default.Table_cddec42a8c1c415ba27927b52d055a08 [id=c60ff09c1e55439cbac9002d52e8d546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.200803 891 tablet_service.cc:1468] Processing CreateTablet for tablet 2f581403ec924fb0948400df41ed8aab (DEFAULT_TABLE table=default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.201210 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a41dbd8782460cb0d3c5f8bd59354f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.201211 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f581403ec924fb0948400df41ed8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.202085 893 tablet_service.cc:1468] Processing CreateTablet for tablet 0edac849e8834894be2d4ff323edd13d (DEFAULT_TABLE table=default.Table_a39be9af778c4fa49c2376cb4bd7397b [id=e9ed839c21da4467a0f165e7fedbe966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.202498 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edac849e8834894be2d4ff323edd13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.204144 1186 tablet_replica.cc:331] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:17.204274 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 0edac849e8834894be2d4ff323edd13d (DEFAULT_TABLE table=default.Table_a39be9af778c4fa49c2376cb4bd7397b [id=e9ed839c21da4467a0f165e7fedbe966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.204341 1186 raft_consensus.cc:2241] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.204620 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edac849e8834894be2d4ff323edd13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.205246 1186 raft_consensus.cc:2270] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.205981 1186 ts_tablet_manager.cc:1905] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:17.209084 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 43111a3512fb4fc0bdf4ee245b16695b (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 43111a3512fb4fc0bdf4ee245b16695b consensus state has no leader
I20250905 19:52:17.209679 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:17.209728 1236 raft_consensus.cc:491] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.209859 1236 raft_consensus.cc:513] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.209960 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d23d6df63c1a4e02a43bccf625e200bb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d23d6df63c1a4e02a43bccf625e200bb already in progress: creating tablet
W20250905 19:52:17.210163 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 757a7d3fba8d43dda36474539b7296ba because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 757a7d3fba8d43dda36474539b7296ba already in progress: creating tablet
W20250905 19:52:17.210338 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a3ae86cb11e24d348e2f12e982f3ab9f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3ae86cb11e24d348e2f12e982f3ab9f already in progress: creating tablet
W20250905 19:52:17.210489 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6d12c7fbb84849c68274810b3fdcba04 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6d12c7fbb84849c68274810b3fdcba04 already in progress: creating tablet
I20250905 19:52:17.210567 1236 leader_election.cc:290] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:17.210700 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 already in progress: creating tablet
I20250905 19:52:17.210964 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59faa6716fed4ea6b563df5f024b2058" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:17.211177 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59faa6716fed4ea6b563df5f024b2058" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:17.211470 984 leader_election.cc:343] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.211705 745 tablet_service.cc:1468] Processing CreateTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f (DEFAULT_TABLE table=default.Table_cddec42a8c1c415ba27927b52d055a08 [id=c60ff09c1e55439cbac9002d52e8d546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.211804 754 tablet_service.cc:1468] Processing CreateTablet for tablet 0edac849e8834894be2d4ff323edd13d (DEFAULT_TABLE table=default.Table_a39be9af778c4fa49c2376cb4bd7397b [id=e9ed839c21da4467a0f165e7fedbe966]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.211828 755 tablet_service.cc:1468] Processing CreateTablet for tablet 2f581403ec924fb0948400df41ed8aab (DEFAULT_TABLE table=default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.214265 1170 tablet_bootstrap.cc:492] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:17.214409 1170 ts_tablet_manager.cc:1397] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.003s
W20250905 19:52:17.215157 986 leader_election.cc:343] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.215173 1170 raft_consensus.cc:357] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.215355 1170 raft_consensus.cc:383] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.215339 986 leader_election.cc:304] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.215417 1170 raft_consensus.cc:738] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.215655 1170 consensus_queue.cc:260] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.212201 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a41dbd8782460cb0d3c5f8bd59354f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.218864 1170 ts_tablet_manager.cc:1428] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:17.220150 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f581403ec924fb0948400df41ed8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.221436 1170 tablet_bootstrap.cc:492] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:17.223534 893 tablet_service.cc:1468] Processing CreateTablet for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (DEFAULT_TABLE table=default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.223915 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9477afb5b47c46d4b24128a6c4bdabc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.224138 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0edac849e8834894be2d4ff323edd13d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.224742 1236 raft_consensus.cc:2747] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.226701 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4204f61929498188c18529148a8d13 (DEFAULT_TABLE table=default.Table_97c63bdbe93a48708207c1e9a96cb1bd [id=fc5c41a92d3e4bfc9c7d07fdd1ec598e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.227130 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4204f61929498188c18529148a8d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.229754 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (DEFAULT_TABLE table=default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.230149 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9477afb5b47c46d4b24128a6c4bdabc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.230304 892 tablet_service.cc:1468] Processing CreateTablet for tablet 2466a1bafc8347e483ca5e28d552bfc9 (DEFAULT_TABLE table=default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.230656 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2466a1bafc8347e483ca5e28d552bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.233266 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2466a1bafc8347e483ca5e28d552bfc9 (DEFAULT_TABLE table=default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.234028 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2466a1bafc8347e483ca5e28d552bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.237051 746 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4204f61929498188c18529148a8d13 (DEFAULT_TABLE table=default.Table_97c63bdbe93a48708207c1e9a96cb1bd [id=fc5c41a92d3e4bfc9c7d07fdd1ec598e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.237464 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4204f61929498188c18529148a8d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.237648 894 tablet_service.cc:1468] Processing CreateTablet for tablet 9f732ae0627c4da5b1691cde9b4d5562 (DEFAULT_TABLE table=default.Table_00ad894476894e1782714d41f84589b3 [id=fe9667d9d9cf4b0e99651b827e50e4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.238126 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f732ae0627c4da5b1691cde9b4d5562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.238927 753 tablet_service.cc:1468] Processing CreateTablet for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (DEFAULT_TABLE table=default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.239288 887 tablet_service.cc:1468] Processing CreateTablet for tablet 3602094b96c14e66b997e8035fc53c40 (DEFAULT_TABLE table=default.Table_10b6351477504574bbcdef1d7f59c24a [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.239753 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3602094b96c14e66b997e8035fc53c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.240033 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9477afb5b47c46d4b24128a6c4bdabc6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.240134 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4c76077e4adf45a7adc736b0e69032eb (DEFAULT_TABLE table=default.Table_48447ad862474825a10962298c97650d [id=d467b5191bf744858924c0ecbbf2e53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.240512 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c76077e4adf45a7adc736b0e69032eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.247608 888 tablet_service.cc:1468] Processing CreateTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f (DEFAULT_TABLE table=default.Table_cddec42a8c1c415ba27927b52d055a08 [id=c60ff09c1e55439cbac9002d52e8d546]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.248085 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 60a41dbd8782460cb0d3c5f8bd59354f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.248476 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3602094b96c14e66b997e8035fc53c40 (DEFAULT_TABLE table=default.Table_10b6351477504574bbcdef1d7f59c24a [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.248526 747 tablet_service.cc:1468] Processing CreateTablet for tablet 9f732ae0627c4da5b1691cde9b4d5562 (DEFAULT_TABLE table=default.Table_00ad894476894e1782714d41f84589b3 [id=fe9667d9d9cf4b0e99651b827e50e4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.249912 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3602094b96c14e66b997e8035fc53c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.250020 748 tablet_service.cc:1468] Processing CreateTablet for tablet 4c76077e4adf45a7adc736b0e69032eb (DEFAULT_TABLE table=default.Table_48447ad862474825a10962298c97650d [id=d467b5191bf744858924c0ecbbf2e53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.261168 891 tablet_service.cc:1468] Processing CreateTablet for tablet 0d4204f61929498188c18529148a8d13 (DEFAULT_TABLE table=default.Table_97c63bdbe93a48708207c1e9a96cb1bd [id=fc5c41a92d3e4bfc9c7d07fdd1ec598e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.261677 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d4204f61929498188c18529148a8d13. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.265560 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c76077e4adf45a7adc736b0e69032eb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.268419 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f732ae0627c4da5b1691cde9b4d5562. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.270800 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 9f732ae0627c4da5b1691cde9b4d5562 (DEFAULT_TABLE table=default.Table_00ad894476894e1782714d41f84589b3 [id=fe9667d9d9cf4b0e99651b827e50e4ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.271335 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f732ae0627c4da5b1691cde9b4d5562. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.271416 440 catalog_manager.cc:5001] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): alter failed for tablet ec810e5cc1af4b9d9f6bd421072bbc77 (table default.Table_2e64b9ef79734139945c53d44ce279e3 [id=89c2ed6b333742bea498aa0da051d60b]),no further retry: Not found: Tablet not found: ec810e5cc1af4b9d9f6bd421072bbc77
I20250905 19:52:17.272617 1170 tablet_bootstrap.cc:654] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.275892 1171 tablet_bootstrap.cc:492] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:17.276073 1171 ts_tablet_manager.cc:1397] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.112s user 0.003s sys 0.001s
I20250905 19:52:17.276085 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 2466a1bafc8347e483ca5e28d552bfc9 (DEFAULT_TABLE table=default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.276475 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2466a1bafc8347e483ca5e28d552bfc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.276891 1171 raft_consensus.cc:357] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.277076 1171 raft_consensus.cc:383] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.277154 1171 raft_consensus.cc:738] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.277968 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3602094b96c14e66b997e8035fc53c40 (DEFAULT_TABLE table=default.Table_10b6351477504574bbcdef1d7f59c24a [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.278290 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3602094b96c14e66b997e8035fc53c40. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.277787 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 2f581403ec924fb0948400df41ed8aab (DEFAULT_TABLE table=default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.281957 746 tablet_service.cc:1468] Processing CreateTablet for tablet 36f966bf512d4cbe9ae406386db31533 (DEFAULT_TABLE table=default.Table_b93afdd4ab954f588fabeb7cfe5713a2 [id=d92c116575544c1c8d98befb5a420348]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.282339 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f966bf512d4cbe9ae406386db31533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.282310 755 tablet_service.cc:1468] Processing CreateTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 (DEFAULT_TABLE table=default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.283044 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01c20eea1b024d7f93f18f0c089efc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.283818 887 tablet_service.cc:1468] Processing CreateTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 (DEFAULT_TABLE table=default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.284287 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01c20eea1b024d7f93f18f0c089efc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.284317 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2f581403ec924fb0948400df41ed8aab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.289275 1171 consensus_queue.cc:260] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.291862 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4c76077e4adf45a7adc736b0e69032eb (DEFAULT_TABLE table=default.Table_48447ad862474825a10962298c97650d [id=d467b5191bf744858924c0ecbbf2e53c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.292294 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4c76077e4adf45a7adc736b0e69032eb. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.293210 441 consensus_peers.cc:489] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 -> Peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Couldn't send request to peer e38333af29db4bf6af41ee5a25d6701e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
W20250905 19:52:17.295167 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 59c13a5b598a4597931ddf5d476aec85 consensus state has no leader
I20250905 19:52:17.295221 892 tablet_service.cc:1468] Processing CreateTablet for tablet 544dffc8b77049c295921d0af091b17d (DEFAULT_TABLE table=default.Table_e24a01de544e4e12a3e017bdb595b067 [id=4044052e369843d39980a31b1654628d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.295476 446 master_service.cc:948] ReplaceTablet: received request to replace tablet 7c5807c1132747e6af79d93e7496d500 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.295552 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 305a7758527d4d52a669b2f2c4ceedbb
I20250905 19:52:17.295619 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 544dffc8b77049c295921d0af091b17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.296132 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 76a5c85dbda140d882a5a63a13aaabd7 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.296873 893 tablet_service.cc:1468] Processing CreateTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 (DEFAULT_TABLE table=default.Table_a2268b9408854caaac5ebd773623563b [id=15dd8b7f1d00449a8f3f25ee97763002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.297185 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 36f966bf512d4cbe9ae406386db31533 (DEFAULT_TABLE table=default.Table_b93afdd4ab954f588fabeb7cfe5713a2 [id=d92c116575544c1c8d98befb5a420348]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.297343 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.297600 1185 tablet_replica.cc:331] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:17.297811 1185 raft_consensus.cc:2241] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.297976 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 (DEFAULT_TABLE table=default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.299616 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.299777 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_315b3dd493ac42fea5c100b3e58d9533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:17.300136 754 tablet_service.cc:1468] Processing CreateTablet for tablet 544dffc8b77049c295921d0af091b17d (DEFAULT_TABLE table=default.Table_e24a01de544e4e12a3e017bdb595b067 [id=4044052e369843d39980a31b1654628d]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:17.300360 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_315b3dd493ac42fea5c100b3e58d9533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.301066 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 544dffc8b77049c295921d0af091b17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.301631 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f966bf512d4cbe9ae406386db31533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.302462 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f2ccd8a7a64b89b44f2d6767ec3466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.304678 1185 raft_consensus.cc:2270] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.305258 1185 ts_tablet_manager.cc:1905] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.306141 753 tablet_service.cc:1468] Processing CreateTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 (DEFAULT_TABLE table=default.Table_a2268b9408854caaac5ebd773623563b [id=15dd8b7f1d00449a8f3f25ee97763002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.306572 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f2ccd8a7a64b89b44f2d6767ec3466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.308436 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 01c20eea1b024d7f93f18f0c089efc42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.309731 894 tablet_service.cc:1468] Processing CreateTablet for tablet 36f966bf512d4cbe9ae406386db31533 (DEFAULT_TABLE table=default.Table_b93afdd4ab954f588fabeb7cfe5713a2 [id=d92c116575544c1c8d98befb5a420348]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.311146 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 36f966bf512d4cbe9ae406386db31533. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.313426 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.314457 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 (DEFAULT_TABLE table=default.Table_a2268b9408854caaac5ebd773623563b [id=15dd8b7f1d00449a8f3f25ee97763002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.314900 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:17.315666 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68f2ccd8a7a64b89b44f2d6767ec3466. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.315647 756 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.315793 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7df10c45b04b4316ad24e8e2d141de9b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.316742 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7df10c45b04b4316ad24e8e2d141de9b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.316867 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.317394 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet e6f837fb7293474b8ad357f52e42e42e
I20250905 19:52:17.319085 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]). Expected version 1 got 0
I20250905 19:52:17.319255 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]). Expected version 1 got 0
I20250905 19:52:17.319505 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 09b46a12076e492e89d0c0178bcf0524
I20250905 19:52:17.320740 1185 ts_tablet_manager.cc:1918] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.320780 756 tablet_service.cc:1515] Processing DeleteTablet for tablet e6f837fb7293474b8ad357f52e42e42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.320868 1185 log.cc:1199] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/305a7758527d4d52a669b2f2c4ceedbb
I20250905 19:52:17.321018 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b46a12076e492e89d0c0178bcf0524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.321220 1185 ts_tablet_manager.cc:1939] T 305a7758527d4d52a669b2f2c4ceedbb P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:17.322000 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b46a12076e492e89d0c0178bcf0524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.322352 890 tablet_service.cc:1515] Processing DeleteTablet for tablet e6f837fb7293474b8ad357f52e42e42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.322508 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b74cc30aa834421da92e129c76067622"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:17.323019 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 305a7758527d4d52a669b2f2c4ceedbb (table default.Table_5996e867324248b1b6c68868f6dfe8e2 [id=2bf888448df6476f818a8efe70480c4b]) successfully deleted
W20250905 19:52:17.323081 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b74cc30aa834421da92e129c76067622 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.323307 1185 tablet_replica.cc:331] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:17.323522 1185 raft_consensus.cc:2241] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.323689 1185 raft_consensus.cc:2270] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.324708 1185 ts_tablet_manager.cc:1905] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.324764 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2e64b9ef79734139945c53d44ce279e3" } new_table_name: "default.Table_28f341e829524e1bb47b0bc5a5e549cd" modify_external_catalogs: true
I20250905 19:52:17.326028 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ccbfb18fea374ff289a19816635f3afd" } new_table_name: "default.Table_aebe460a114e490ab8dfed41377a96b8" modify_external_catalogs: true
I20250905 19:52:17.327714 1171 ts_tablet_manager.cc:1428] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.052s user 0.000s sys 0.002s
I20250905 19:52:17.328302 1236 raft_consensus.cc:491] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.328449 1236 raft_consensus.cc:513] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.328651 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 59faa6716fed4ea6b563df5f024b2058
I20250905 19:52:17.329130 451 catalog_manager.cc:6652] ReplaceTablet: tablet 59faa6716fed4ea6b563df5f024b2058 deleted and replaced by tablet 05ec559d0f83445892f8285f9bdeb2c8
I20250905 19:52:17.329125 1236 leader_election.cc:290] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.329475 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b46a12076e492e89d0c0178bcf0524" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.329754 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09b46a12076e492e89d0c0178bcf0524" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.329973 986 leader_election.cc:343] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: Tablet not RUNNING: STOPPED
I20250905 19:52:17.330201 453 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca] alter complete (version 1)
W20250905 19:52:17.330268 984 leader_election.cc:343] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.330432 984 leader_election.cc:304] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.332247 1236 raft_consensus.cc:2747] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.332337 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 59faa6716fed4ea6b563df5f024b2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 05ec559d0f83445892f8285f9bdeb2c8) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.332760 1171 tablet_bootstrap.cc:492] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:17.333583 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet e6f837fb7293474b8ad357f52e42e42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.334017 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_48c2edbf02444d7580fff76f96c9c655"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.334578 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_48c2edbf02444d7580fff76f96c9c655 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.334640 1171 tablet_bootstrap.cc:654] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.336169 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_58b0d391a4044d39951ca7f9799bcf70" } new_table_name: "default.Table_9ae350716fe144629f1f50ebee119d72" modify_external_catalogs: true
I20250905 19:52:17.336403 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 59faa6716fed4ea6b563df5f024b2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 05ec559d0f83445892f8285f9bdeb2c8) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.337360 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a1eef46eeeda4dc492c96d6ee0dab2ae"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.337872 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a1eef46eeeda4dc492c96d6ee0dab2ae in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.338702 1236 raft_consensus.cc:491] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.338853 1236 raft_consensus.cc:513] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.339530 1236 leader_election.cc:290] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.339749 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 0edac849e8834894be2d4ff323edd13d
I20250905 19:52:17.340140 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d20e269bf2224fe49f701c4777c42632" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.340659 984 leader_election.cc:343] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.341193 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.341221 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d20e269bf2224fe49f701c4777c42632" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.341442 914 raft_consensus.cc:2466] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:17.341863 986 leader_election.cc:304] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.341966 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8e07fab3c1d94464a864b8aa58a9372d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.342514 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e07fab3c1d94464a864b8aa58a9372d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.342630 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:17.342684 1236 raft_consensus.cc:2802] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:17.342782 1236 raft_consensus.cc:491] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:17.342862 1236 raft_consensus.cc:3058] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.343458 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_32577b5aebfd4f37b4400d62e431d7f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.344040 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_32577b5aebfd4f37b4400d62e431d7f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.345691 1236 raft_consensus.cc:513] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.346722 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d20e269bf2224fe49f701c4777c42632" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:17.347703 984 leader_election.cc:343] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.348570 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d20e269bf2224fe49f701c4777c42632" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:17.348793 914 raft_consensus.cc:3058] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.351495 914 raft_consensus.cc:2466] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:17.352738 892 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.354142 749 tablet_service.cc:1468] Processing CreateTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc (DEFAULT_TABLE table=default.Table_ccbfb18fea374ff289a19816635f3afd [id=f03c03ae24744485b3ac71752d70f7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.354969 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.355237 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74a7e7d9874e4d26bda3ab24a6e24adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.357090 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 76a5c85dbda140d882a5a63a13aaabd7
I20250905 19:52:17.357599 455 catalog_manager.cc:6652] ReplaceTablet: tablet 76a5c85dbda140d882a5a63a13aaabd7 deleted and replaced by tablet f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:17.358194 888 tablet_service.cc:1468] Processing CreateTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc (DEFAULT_TABLE table=default.Table_ccbfb18fea374ff289a19816635f3afd [id=f03c03ae24744485b3ac71752d70f7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.358575 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74a7e7d9874e4d26bda3ab24a6e24adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.360900 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 with delete_type TABLET_DATA_DELETED (replaced by tablet f28e403dc2f6471ebea1805fd843cfb1) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.366839 1186 ts_tablet_manager.cc:1918] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.367069 1186 log.cc:1199] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4f1d960461ef4829960baa5cca7f7765
I20250905 19:52:17.367486 1186 ts_tablet_manager.cc:1939] T 4f1d960461ef4829960baa5cca7f7765 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:17.368091 1186 tablet_replica.cc:331] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:17.368856 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 77cbdda0e2d641bd8092f00925bb9a9c from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.369460 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9a006790dc8f41cf8cb1e4ffc3d77175"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.370034 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.370287 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4f1d960461ef4829960baa5cca7f7765 (table default.Table_b550173c58174bedacdcb184e73e8e9c [id=c56b6d4d3b204b0981bb9cecd8a1a08c]) successfully deleted
W20250905 19:52:17.371922 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 74a7e7d9874e4d26bda3ab24a6e24adc consensus state has no leader
I20250905 19:52:17.372264 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 with delete_type TABLET_DATA_DELETED (replaced by tablet f28e403dc2f6471ebea1805fd843cfb1) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:17.372658 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9477afb5b47c46d4b24128a6c4bdabc6 consensus state has no leader
W20250905 19:52:17.377138 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 00d8144bb4f14358af539b7aae326918 has no consensus state
I20250905 19:52:17.379493 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_20b48d10bd25499c9543b27963456363"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.380223 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_20b48d10bd25499c9543b27963456363 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.380517 1185 ts_tablet_manager.cc:1918] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.380661 1185 log.cc:1199] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/09b46a12076e492e89d0c0178bcf0524
I20250905 19:52:17.382117 986 leader_election.cc:304] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.382730 1236 leader_election.cc:290] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.383112 1381 raft_consensus.cc:2802] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:17.383280 1381 raft_consensus.cc:695] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:17.383570 1381 consensus_queue.cc:237] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.385591 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.385958 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d7567807594201a598d715ffbb318b with delete_type TABLET_DATA_DELETED (replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.386418 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc (DEFAULT_TABLE table=default.Table_ccbfb18fea374ff289a19816635f3afd [id=f03c03ae24744485b3ac71752d70f7d8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.386873 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.387437 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 544dffc8b77049c295921d0af091b17d (DEFAULT_TABLE table=default.Table_e24a01de544e4e12a3e017bdb595b067 [id=4044052e369843d39980a31b1654628d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.388209 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 59faa6716fed4ea6b563df5f024b2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 05ec559d0f83445892f8285f9bdeb2c8) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.389250 1184 tablet_replica.cc:331] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:17.389443 1186 raft_consensus.cc:2241] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:17.390098 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74a7e7d9874e4d26bda3ab24a6e24adc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.376951 1203 raft_consensus.cc:491] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.391430 1203 raft_consensus.cc:513] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.392161 1203 leader_election.cc:290] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:17.393138 711 leader_election.cc:343] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
W20250905 19:52:17.394187 712 leader_election.cc:343] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 6d09371944644b3082268b2b37ce54f8
I20250905 19:52:17.394369 712 leader_election.cc:304] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:17.394654 1203 raft_consensus.cc:2747] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.386157 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d09371944644b3082268b2b37ce54f8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:14 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.370092 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f4ec474911941b2a2d0825cdb37bd72 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.395736 1186 raft_consensus.cc:2270] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.397826 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 544dffc8b77049c295921d0af091b17d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.397902 447 master_service.cc:948] ReplaceTablet: received request to replace tablet a8a32f742eaf473e9f6f9bf214f53679 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.398104 1186 ts_tablet_manager.cc:1905] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.399946 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 with delete_type TABLET_DATA_DELETED (replaced by tablet f28e403dc2f6471ebea1805fd843cfb1) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.401140 1184 raft_consensus.cc:2241] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.401621 1184 raft_consensus.cc:2270] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.402639 1184 ts_tablet_manager.cc:1905] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.406466 1170 tablet_bootstrap.cc:492] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:17.406702 1170 ts_tablet_manager.cc:1397] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.185s user 0.002s sys 0.005s
I20250905 19:52:17.407581 1170 raft_consensus.cc:357] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.407902 1170 raft_consensus.cc:383] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.408061 1170 raft_consensus.cc:738] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.408460 1170 consensus_queue.cc:260] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.409173 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a1eef46eeeda4dc492c96d6ee0dab2ae" } new_table_name: "default.Table_ab2b806012994ad38e6f7a3171f00777" modify_external_catalogs: true
I20250905 19:52:17.408737 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b46a12076e492e89d0c0178bcf0524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.412253 1170 ts_tablet_manager.cc:1428] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:17.412700 1169 tablet_bootstrap.cc:492] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.412886 1169 ts_tablet_manager.cc:1397] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.222s user 0.027s sys 0.050s
I20250905 19:52:17.415017 1170 tablet_bootstrap.cc:492] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:17.417416 1170 tablet_bootstrap.cc:654] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.417526 1169 raft_consensus.cc:357] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.417721 1169 raft_consensus.cc:383] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.417805 1169 raft_consensus.cc:738] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.418071 1169 consensus_queue.cc:260] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.421375 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05ae83eea5f74e9187713e8b82c9a23b has no consensus state
I20250905 19:52:17.422305 1186 ts_tablet_manager.cc:1918] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:17.414719 1171 tablet_bootstrap.cc:492] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:17.423002 1171 ts_tablet_manager.cc:1397] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.090s user 0.004s sys 0.000s
I20250905 19:52:17.423895 1171 raft_consensus.cc:357] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.424157 1171 raft_consensus.cc:383] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.424237 1171 raft_consensus.cc:738] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.425034 1171 consensus_queue.cc:260] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.428647 455 catalog_manager.cc:5582] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:17.428951 1169 ts_tablet_manager.cc:1428] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250905 19:52:17.429087 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 2 got 0
I20250905 19:52:17.429313 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_3f876cdc8a6e4380a81a2b4d49cfb138 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 1 got 0
I20250905 19:52:17.434161 1169 tablet_bootstrap.cc:492] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.435566 893 tablet_service.cc:1468] Processing CreateTablet for tablet 4fe94d1254d04949aa25346d8e323c3b (DEFAULT_TABLE table=default.Table_32577b5aebfd4f37b4400d62e431d7f1 [id=e0eba87919924759a4a5beae73fd3ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.435804 1169 tablet_bootstrap.cc:654] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.436049 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe94d1254d04949aa25346d8e323c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.436580 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:17.436743 1185 ts_tablet_manager.cc:1939] T 09b46a12076e492e89d0c0178bcf0524 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:17.437052 447 catalog_manager.cc:6652] ReplaceTablet: tablet a8a32f742eaf473e9f6f9bf214f53679 deleted and replaced by tablet 723168b075874dfcb3eb50c5134c499e
I20250905 19:52:17.439487 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e113565700b647959f0e06bc381630e6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.440133 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e113565700b647959f0e06bc381630e6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.444132 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.445075 1184 ts_tablet_manager.cc:1918] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.445210 1184 log.cc:1199] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/305a7758527d4d52a669b2f2c4ceedbb
I20250905 19:52:17.438488 753 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.445595 1184 ts_tablet_manager.cc:1939] T 305a7758527d4d52a669b2f2c4ceedbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:17.445961 1186 log.cc:1199] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/266ffb862d144368994f5c33e684647d
I20250905 19:52:17.446427 1186 ts_tablet_manager.cc:1939] T 266ffb862d144368994f5c33e684647d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:17.447219 1184 tablet_replica.cc:331] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:17.447433 1184 raft_consensus.cc:2241] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.447597 1184 raft_consensus.cc:2270] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.448184 1184 ts_tablet_manager.cc:1905] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.449009 894 tablet_service.cc:1468] Processing CreateTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea (DEFAULT_TABLE table=default.Table_7df10c45b04b4316ad24e8e2d141de9b [id=8f60da23a0864f92ba7c7ffcfa65277f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.449402 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262bcd1cfc994bb58e999b141b9bc7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.449558 1381 raft_consensus.cc:491] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.449740 1381 raft_consensus.cc:513] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.450203 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 305a7758527d4d52a669b2f2c4ceedbb (table default.Table_5996e867324248b1b6c68868f6dfe8e2 [id=2bf888448df6476f818a8efe70480c4b]) successfully deleted
W20250905 19:52:17.450510 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet d480531ba94f4b0f9fe40e88fe91512e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d480531ba94f4b0f9fe40e88fe91512e already in progress: creating tablet
W20250905 19:52:17.450718 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet e6f837fb7293474b8ad357f52e42e42e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6f837fb7293474b8ad357f52e42e42e already in progress: creating tablet
I20250905 19:52:17.450896 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e421213b7bd4dd8a03515519395de7d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.451443 984 leader_election.cc:343] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.451514 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 (DEFAULT_TABLE table=default.Table_8e07fab3c1d94464a864b8aa58a9372d [id=bbdef6ec6956464f9cdd00b3fb506c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.451632 1381 leader_election.cc:290] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.451800 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e421213b7bd4dd8a03515519395de7d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.452015 914 raft_consensus.cc:2466] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:17.452055 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea (DEFAULT_TABLE table=default.Table_7df10c45b04b4316ad24e8e2d141de9b [id=8f60da23a0864f92ba7c7ffcfa65277f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.452479 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 (DEFAULT_TABLE table=default.Table_48c2edbf02444d7580fff76f96c9c655 [id=12633ed634a44d74830d55d4e7191808]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.452558 892 tablet_service.cc:1468] Processing CreateTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 (DEFAULT_TABLE table=default.Table_48c2edbf02444d7580fff76f96c9c655 [id=12633ed634a44d74830d55d4e7191808]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.452916 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 4fe94d1254d04949aa25346d8e323c3b (DEFAULT_TABLE table=default.Table_32577b5aebfd4f37b4400d62e431d7f1 [id=e0eba87919924759a4a5beae73fd3ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.452991 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d84653c9e7a4857a7580fa85f8e4732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.453002 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 266ffb862d144368994f5c33e684647d (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]) successfully deleted
W20250905 19:52:17.453222 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 305a7758527d4d52a669b2f2c4ceedbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 305a7758527d4d52a669b2f2c4ceedbb already in progress: creating tablet
I20250905 19:52:17.453977 753 tablet_service.cc:1468] Processing CreateTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 (DEFAULT_TABLE table=default.Table_8e07fab3c1d94464a864b8aa58a9372d [id=bbdef6ec6956464f9cdd00b3fb506c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.454191 750 tablet_service.cc:1468] Processing CreateTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f (DEFAULT_TABLE table=default.Table_315b3dd493ac42fea5c100b3e58d9533 [id=70f8481033284889b47c81173fb2ef5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.454440 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c2b030fafd44ad9e648b2527a4ad07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.454730 754 tablet_service.cc:1468] Processing CreateTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea (DEFAULT_TABLE table=default.Table_7df10c45b04b4316ad24e8e2d141de9b [id=8f60da23a0864f92ba7c7ffcfa65277f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.455073 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262bcd1cfc994bb58e999b141b9bc7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.460296 746 tablet_service.cc:1468] Processing CreateTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 (DEFAULT_TABLE table=default.Table_48c2edbf02444d7580fff76f96c9c655 [id=12633ed634a44d74830d55d4e7191808]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.460727 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d84653c9e7a4857a7580fa85f8e4732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.461092 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c2b030fafd44ad9e648b2527a4ad07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.461282 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.461648 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 262bcd1cfc994bb58e999b141b9bc7ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.462175 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2d84653c9e7a4857a7580fa85f8e4732. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.464854 986 leader_election.cc:304] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.466419 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.466697 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 1664cbe248514535953a1ef2d312f98f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.467208 893 tablet_service.cc:1468] Processing CreateTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 (DEFAULT_TABLE table=default.Table_8e07fab3c1d94464a864b8aa58a9372d [id=bbdef6ec6956464f9cdd00b3fb506c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.468343 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29c2b030fafd44ad9e648b2527a4ad07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.469914 1236 raft_consensus.cc:2802] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:17.470029 1236 raft_consensus.cc:491] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:17.470289 1236 raft_consensus.cc:3058] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.470710 1186 tablet_replica.cc:331] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:17.470896 1186 raft_consensus.cc:2241] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.471592 1186 raft_consensus.cc:2270] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.471647 1381 raft_consensus.cc:491] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.471802 1381 raft_consensus.cc:513] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.472045 891 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.473091 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe94d1254d04949aa25346d8e323c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.473416 455 catalog_manager.cc:5582] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:17.473946 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]). Expected version 1 got 0
I20250905 19:52:17.475672 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7d7567807594201a598d715ffbb318b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.475879 914 raft_consensus.cc:2466] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:17.476487 986 leader_election.cc:304] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:17.477176 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d480531ba94f4b0f9fe40e88fe91512e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d480531ba94f4b0f9fe40e88fe91512e already in progress: creating tablet
W20250905 19:52:17.477434 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e6f837fb7293474b8ad357f52e42e42e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6f837fb7293474b8ad357f52e42e42e already in progress: creating tablet
W20250905 19:52:17.477632 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 09b46a12076e492e89d0c0178bcf0524 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 09b46a12076e492e89d0c0178bcf0524 already in progress: creating tablet
W20250905 19:52:17.477851 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 59faa6716fed4ea6b563df5f024b2058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59faa6716fed4ea6b563df5f024b2058 already in progress: creating tablet
W20250905 19:52:17.478065 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0edac849e8834894be2d4ff323edd13d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0edac849e8834894be2d4ff323edd13d already in progress: creating tablet
W20250905 19:52:17.478291 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 76a5c85dbda140d882a5a63a13aaabd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76a5c85dbda140d882a5a63a13aaabd7 already in progress: creating tablet
I20250905 19:52:17.478427 1236 raft_consensus.cc:513] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.479918 1176 raft_consensus.cc:2802] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:17.480041 1176 raft_consensus.cc:491] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:17.480182 1176 raft_consensus.cc:3058] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.482203 1381 leader_election.cc:290] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.473819 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7d7567807594201a598d715ffbb318b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.484809 984 leader_election.cc:343] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.484871 1176 raft_consensus.cc:513] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.479565 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e421213b7bd4dd8a03515519395de7d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:17.485682 1176 leader_election.cc:290] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.486578 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 09b46a12076e492e89d0c0178bcf0524 (table default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a [id=3b8967eb30ad46cdbf272136f23318f1]) successfully deleted
I20250905 19:52:17.486899 892 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d7567807594201a598d715ffbb318b with delete_type TABLET_DATA_DELETED (replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:17.487295 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet e6f837fb7293474b8ad357f52e42e42e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet e6f837fb7293474b8ad357f52e42e42e already in progress: creating tablet
I20250905 19:52:17.487388 1171 ts_tablet_manager.cc:1428] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.064s user 0.001s sys 0.001s
I20250905 19:52:17.487390 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7d7567807594201a598d715ffbb318b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:17.487747 1171 tablet_bootstrap.cc:492] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:17.487857 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 59faa6716fed4ea6b563df5f024b2058 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 59faa6716fed4ea6b563df5f024b2058 already in progress: creating tablet
W20250905 19:52:17.488723 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 0edac849e8834894be2d4ff323edd13d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0edac849e8834894be2d4ff323edd13d already in progress: creating tablet
W20250905 19:52:17.489763 984 leader_election.cc:343] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:17.490123 984 leader_election.cc:343] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.481220 1170 tablet_bootstrap.cc:492] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:17.481801 755 tablet_service.cc:1468] Processing CreateTablet for tablet 4fe94d1254d04949aa25346d8e323c3b (DEFAULT_TABLE table=default.Table_32577b5aebfd4f37b4400d62e431d7f1 [id=e0eba87919924759a4a5beae73fd3ac9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.491230 1170 ts_tablet_manager.cc:1397] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.000s
I20250905 19:52:17.491362 1236 leader_election.cc:290] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:17.491597 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4fe94d1254d04949aa25346d8e323c3b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.492120 1170 raft_consensus.cc:357] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.492316 1170 raft_consensus.cc:383] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.492398 1170 raft_consensus.cc:738] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.492658 1170 consensus_queue.cc:260] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.493212 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet d480531ba94f4b0f9fe40e88fe91512e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d480531ba94f4b0f9fe40e88fe91512e already in progress: creating tablet
W20250905 19:52:17.493912 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 76a5c85dbda140d882a5a63a13aaabd7 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 76a5c85dbda140d882a5a63a13aaabd7 already in progress: creating tablet
I20250905 19:52:17.490875 1381 consensus_queue.cc:1035] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.012s
W20250905 19:52:17.494557 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a8a32f742eaf473e9f6f9bf214f53679 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8a32f742eaf473e9f6f9bf214f53679 already in progress: creating tablet
I20250905 19:52:17.494941 1185 tablet_replica.cc:331] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:17.495185 1185 raft_consensus.cc:2241] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.495411 1185 raft_consensus.cc:2270] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.498168 1170 ts_tablet_manager.cc:1428] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 19:52:17.496881 891 tablet_service.cc:1468] Processing CreateTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f (DEFAULT_TABLE table=default.Table_315b3dd493ac42fea5c100b3e58d9533 [id=70f8481033284889b47c81173fb2ef5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.497579 894 tablet_service.cc:1468] Processing CreateTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 (DEFAULT_TABLE table=default.Table_46df29d8bee2462fb784be95829c6ad0 [id=3ed507a38d1049fa893efa7be65f3609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.496347 1185 ts_tablet_manager.cc:1905] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.499234 754 tablet_service.cc:1468] Processing CreateTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 (DEFAULT_TABLE table=default.Table_46df29d8bee2462fb784be95829c6ad0 [id=3ed507a38d1049fa893efa7be65f3609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.499622 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0702063ec1c4f6fba58e6d4f3912212. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.500944 1170 tablet_bootstrap.cc:492] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:17.503161 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b93afdd4ab954f588fabeb7cfe5713a2" } new_table_name: "default.Table_51169c758cd646cd91f3bda396598e10" modify_external_catalogs: true
I20250905 19:52:17.503296 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 78ac5c7e9b6741d79db2e606d55a3509 with delete_type TABLET_DATA_DELETED (replaced by tablet 59c13a5b598a4597931ddf5d476aec85) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.508924 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e421213b7bd4dd8a03515519395de7d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:17.509127 913 raft_consensus.cc:3058] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.511649 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_16b3928f208649fa879486e9d7d6a235" } modify_external_catalogs: true
I20250905 19:52:17.511785 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.511862 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_16b3928f208649fa879486e9d7d6a235" } modify_external_catalogs: true
W20250905 19:52:17.512624 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 36f966bf512d4cbe9ae406386db31533 consensus state has no leader
I20250905 19:52:17.512609 913 raft_consensus.cc:2466] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:17.513561 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7d7567807594201a598d715ffbb318b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:17.516307 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 78ac5c7e9b6741d79db2e606d55a3509 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.517551 986 leader_election.cc:304] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:17.519196 986 leader_election.cc:343] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: Tablet not RUNNING: STOPPED
I20250905 19:52:17.518085 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f (DEFAULT_TABLE table=default.Table_315b3dd493ac42fea5c100b3e58d9533 [id=70f8481033284889b47c81173fb2ef5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.520203 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.521262 750 tablet_service.cc:1468] Processing CreateTablet for tablet 7cdd4de055864d4fa6adb36bf190e609 (DEFAULT_TABLE table=default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.521644 888 tablet_service.cc:1468] Processing CreateTablet for tablet 7cdd4de055864d4fa6adb36bf190e609 (DEFAULT_TABLE table=default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.522032 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdd4de055864d4fa6adb36bf190e609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.522207 1236 raft_consensus.cc:491] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.522436 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdd4de055864d4fa6adb36bf190e609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.522377 1236 raft_consensus.cc:513] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.522982 1236 leader_election.cc:290] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.523149 1236 raft_consensus.cc:2802] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:17.523777 1236 raft_consensus.cc:695] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:17.523747 746 tablet_service.cc:1468] Processing CreateTablet for tablet 8feaf4ec6b0e4ce9909dd72258912935 (DEFAULT_TABLE table=default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.524130 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8feaf4ec6b0e4ce9909dd72258912935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.525007 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 7aa06165136a4fa5a935dfdd786a75c4
I20250905 19:52:17.525003 745 tablet_service.cc:1468] Processing CreateTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 (DEFAULT_TABLE table=default.Table_9f1863dcd87d4fb98b0f2e89677b5edf [id=d66e311f2eb84a6e889e748c5ee5f7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.525364 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a81376d4c0b47a9817deae4d34ffb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.525316 986 leader_election.cc:304] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.527043 1176 raft_consensus.cc:2747] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.528914 753 tablet_service.cc:1468] Processing CreateTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 (DEFAULT_TABLE table=default.Table_5918e71853e2403a9d36c683ed58ac88 [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.529012 774 raft_consensus.cc:3058] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.529253 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302053e1611b4e22a9a2e7b60ae68f27. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.530686 984 consensus_peers.cc:489] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:17.530874 1176 raft_consensus.cc:491] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.530990 1176 raft_consensus.cc:513] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.531586 1176 leader_election.cc:290] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.532199 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c884954e5dc4445a12252953593d0bc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:17.532310 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf2881d228964cd3bef040a06045d8d2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.532716 984 leader_election.cc:343] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:17.532953 984 leader_election.cc:343] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.533154 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fe362aa06bd64814853c72f1a41c76b3" } modify_external_catalogs: true
I20250905 19:52:17.533327 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fe362aa06bd64814853c72f1a41c76b3" } modify_external_catalogs: true
I20250905 19:52:17.533759 1028 tablet_service.cc:1468] Processing CreateTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 (DEFAULT_TABLE table=default.Table_46df29d8bee2462fb784be95829c6ad0 [id=3ed507a38d1049fa893efa7be65f3609]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.534159 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0702063ec1c4f6fba58e6d4f3912212. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.522519 887 tablet_service.cc:1468] Processing CreateTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 (DEFAULT_TABLE table=default.Table_5918e71853e2403a9d36c683ed58ac88 [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.524070 890 tablet_service.cc:1468] Processing CreateTablet for tablet 8feaf4ec6b0e4ce9909dd72258912935 (DEFAULT_TABLE table=default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.525632 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 7aa06165136a4fa5a935dfdd786a75c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.527823 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf2881d228964cd3bef040a06045d8d2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.536150 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302053e1611b4e22a9a2e7b60ae68f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.536513 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 7cdd4de055864d4fa6adb36bf190e609 (DEFAULT_TABLE table=default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.537026 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cdd4de055864d4fa6adb36bf190e609. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.540103 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8feaf4ec6b0e4ce9909dd72258912935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.540547 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 17d0a48c25b84facb082670cb898468c
I20250905 19:52:17.529799 1171 tablet_bootstrap.cc:654] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.541579 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d0a48c25b84facb082670cb898468c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.543496 748 tablet_service.cc:1468] Processing CreateTablet for tablet 78076f6534814939ae8c2db9a4bd1879 (DEFAULT_TABLE table=default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.543956 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78076f6534814939ae8c2db9a4bd1879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.544610 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 8feaf4ec6b0e4ce9909dd72258912935 (DEFAULT_TABLE table=default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.545008 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8feaf4ec6b0e4ce9909dd72258912935. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.545059 1381 consensus_queue.cc:1035] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:17.548193 1236 consensus_queue.cc:237] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.548591 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 7aa06165136a4fa5a935dfdd786a75c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.550179 755 tablet_service.cc:1468] Processing CreateTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab (DEFAULT_TABLE table=default.Table_7240f6ab8dbf4c16a2651dfc5a92d15b [id=711f36b2be7c4a368fcb3106e691b133]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.550774 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d0a48c25b84facb082670cb898468c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.551461 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed6109bdf8f4caf8c223a42401b94ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.551631 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 (DEFAULT_TABLE table=default.Table_9f1863dcd87d4fb98b0f2e89677b5edf [id=d66e311f2eb84a6e889e748c5ee5f7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.554010 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ab2b806012994ad38e6f7a3171f00777" } modify_external_catalogs: true
I20250905 19:52:17.554194 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 7aa06165136a4fa5a935dfdd786a75c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.554273 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ab2b806012994ad38e6f7a3171f00777" } modify_external_catalogs: true
I20250905 19:52:17.555416 1169 tablet_bootstrap.cc:492] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.555683 1169 ts_tablet_manager.cc:1397] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.122s user 0.000s sys 0.008s
I20250905 19:52:17.556739 1169 raft_consensus.cc:357] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.557018 1169 raft_consensus.cc:383] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.556779 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_46df29d8bee2462fb784be95829c6ad0" } modify_external_catalogs: true
I20250905 19:52:17.557228 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_46df29d8bee2462fb784be95829c6ad0" } modify_external_catalogs: true
I20250905 19:52:17.557477 1169 raft_consensus.cc:738] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.557823 1169 consensus_queue.cc:260] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.558087 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab (DEFAULT_TABLE table=default.Table_7240f6ab8dbf4c16a2651dfc5a92d15b [id=711f36b2be7c4a368fcb3106e691b133]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.558317 893 tablet_service.cc:1468] Processing CreateTablet for tablet 78076f6534814939ae8c2db9a4bd1879 (DEFAULT_TABLE table=default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.558426 1169 ts_tablet_manager.cc:1428] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:17.558706 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b0702063ec1c4f6fba58e6d4f3912212. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.558723 1169 tablet_bootstrap.cc:492] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.560438 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 17d0a48c25b84facb082670cb898468c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.561231 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78076f6534814939ae8c2db9a4bd1879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.561700 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 (DEFAULT_TABLE table=default.Table_5918e71853e2403a9d36c683ed58ac88 [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.564553 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 05ae83eea5f74e9187713e8b82c9a23b
I20250905 19:52:17.562155 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 (DEFAULT_TABLE table=default.Table_9f1863dcd87d4fb98b0f2e89677b5edf [id=d66e311f2eb84a6e889e748c5ee5f7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.564043 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 78076f6534814939ae8c2db9a4bd1879 (DEFAULT_TABLE table=default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.568445 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 78076f6534814939ae8c2db9a4bd1879. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.564898 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab (DEFAULT_TABLE table=default.Table_7240f6ab8dbf4c16a2651dfc5a92d15b [id=711f36b2be7c4a368fcb3106e691b133]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.572454 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed6109bdf8f4caf8c223a42401b94ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.574510 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a81376d4c0b47a9817deae4d34ffb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.576838 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:17.577391 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8ed6109bdf8f4caf8c223a42401b94ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.581120 753 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.557936 1170 tablet_bootstrap.cc:654] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.582399 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.583349 884 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.583695 885 tablet_service.cc:1468] Processing CreateTablet for tablet 0039b8f0ba704907b2820c9e3a69cd73 (DEFAULT_TABLE table=default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.584163 914 raft_consensus.cc:1273] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Refusing update from remote peer 4eb3af79b01d4835bf6bf7b26bf016f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:17.584225 455 catalog_manager.cc:5582] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:17.584869 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a81376d4c0b47a9817deae4d34ffb47. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.585775 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0039b8f0ba704907b2820c9e3a69cd73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.590842 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_aebe460a114e490ab8dfed41377a96b8" } modify_external_catalogs: true
I20250905 19:52:17.591081 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_aebe460a114e490ab8dfed41377a96b8" } modify_external_catalogs: true
I20250905 19:52:17.591143 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4bdeb93947c346b1a36458d0dd5f5d2a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.591725 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4bdeb93947c346b1a36458d0dd5f5d2a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.592630 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302053e1611b4e22a9a2e7b60ae68f27. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.596225 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 78076f6534814939ae8c2db9a4bd1879 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:17.597025 1169 tablet_bootstrap.cc:654] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:17.599114 986 leader_election.cc:343] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.599264 986 leader_election.cc:304] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.602092 1381 raft_consensus.cc:2747] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.602155 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_33489afbbd2a47cfa0eaea7cfcba8f17" } new_table_name: "default.Table_0c5be34d0e4548f1b16cffbd509c6037" modify_external_catalogs: true
I20250905 19:52:17.602353 1381 consensus_queue.cc:1035] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:17.603559 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_10b6351477504574bbcdef1d7f59c24a" } new_table_name: "default.Table_e2b7277d0ccd4d0aa90b08e13f6eb572" modify_external_catalogs: true
I20250905 19:52:17.605237 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_906a087303e14db48ac1470f5c6953bd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.605773 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_906a087303e14db48ac1470f5c6953bd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.607028 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_97197831cd2f45cc909b9d7ab0caebc2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.607555 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_97197831cd2f45cc909b9d7ab0caebc2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.608656 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_2d5b20700ad74baf8138bdcadb0bef37"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.609124 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2d5b20700ad74baf8138bdcadb0bef37 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.610226 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e24a01de544e4e12a3e017bdb595b067" } new_table_name: "default.Table_1f75eecc8c8941a6aa9982f4e8cc6450" modify_external_catalogs: true
I20250905 19:52:17.612176 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.612617 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.613240 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a19dd25caac94580b689031eda61e7e4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.613729 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a19dd25caac94580b689031eda61e7e4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.616464 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 0039b8f0ba704907b2820c9e3a69cd73 (DEFAULT_TABLE table=default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.616959 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0039b8f0ba704907b2820c9e3a69cd73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.618031 746 tablet_service.cc:1468] Processing CreateTablet for tablet a5f076d9917f40d899d7422ad974fe0f (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.618904 750 tablet_service.cc:1468] Processing CreateTablet for tablet 6e89d23c4cb64f568b25f92a278b9d5e (DEFAULT_TABLE table=default.Table_b74cc30aa834421da92e129c76067622 [id=853047bedbb242d893b5be8b5a5c156e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.619434 745 tablet_service.cc:1468] Processing CreateTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 (DEFAULT_TABLE table=default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.620057 753 tablet_service.cc:1468] Processing CreateTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b (DEFAULT_TABLE table=default.Table_ab2b806012994ad38e6f7a3171f00777 [id=29f567c56184458c96d24eb78f402137]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.620461 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f076d9917f40d899d7422ad974fe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.621627 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28e403dc2f6471ebea1805fd843cfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.621951 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:17.622390 447 catalog_manager.cc:6652] ReplaceTablet: tablet 78076f6534814939ae8c2db9a4bd1879 deleted and replaced by tablet 53ac46ce05694bc2b53e195638be9e23
I20250905 19:52:17.622697 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.622763 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_834cb637abd74d7694ba9e3f8e4ed84a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:17.623210 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_3fa477c60f534a63b2cd4e54fe7c5c40"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.623311 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_834cb637abd74d7694ba9e3f8e4ed84a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.560731 1186 ts_tablet_manager.cc:1905] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.565253 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.624507 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e89d23c4cb64f568b25f92a278b9d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.625174 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6e89d23c4cb64f568b25f92a278b9d5e (DEFAULT_TABLE table=default.Table_b74cc30aa834421da92e129c76067622 [id=853047bedbb242d893b5be8b5a5c156e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.625540 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e89d23c4cb64f568b25f92a278b9d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.628207 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ae83eea5f74e9187713e8b82c9a23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.631142 1186 ts_tablet_manager.cc:1918] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.631269 1186 log.cc:1199] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6d09371944644b3082268b2b37ce54f8
I20250905 19:52:17.631651 1186 ts_tablet_manager.cc:1939] T 6d09371944644b3082268b2b37ce54f8 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:17.631951 1024 tablet_service.cc:1468] Processing CreateTablet for tablet a5f076d9917f40d899d7422ad974fe0f (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.632547 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f076d9917f40d899d7422ad974fe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.567265 754 tablet_service.cc:1468] Processing CreateTablet for tablet 0039b8f0ba704907b2820c9e3a69cd73 (DEFAULT_TABLE table=default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.637251 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c884954e5dc4445a12252953593d0bc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:17.642411 848 consensus_peers.cc:489] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:17.642686 1184 ts_tablet_manager.cc:1918] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.642938 1184 log.cc:1199] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/f8d3cbdda0c94e139259d90919dc2f95
I20250905 19:52:17.643436 1184 ts_tablet_manager.cc:1939] T f8d3cbdda0c94e139259d90919dc2f95 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:17.644240 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 74a7e7d9874e4d26bda3ab24a6e24adc
I20250905 19:52:17.644495 1184 tablet_replica.cc:331] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:17.644730 1184 raft_consensus.cc:2241] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.644979 1184 raft_consensus.cc:2270] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.645526 1185 ts_tablet_manager.cc:1918] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.645668 1185 log.cc:1199] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d7d7567807594201a598d715ffbb318b
I20250905 19:52:17.645911 1184 ts_tablet_manager.cc:1905] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.646207 1185 ts_tablet_manager.cc:1939] T d7d7567807594201a598d715ffbb318b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:17.646466 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3fa477c60f534a63b2cd4e54fe7c5c40 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.647112 1185 tablet_replica.cc:331] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:17.648222 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3602094b96c14e66b997e8035fc53c40 consensus state has no leader
W20250905 19:52:17.649700 986 leader_election.cc:343] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.649860 986 leader_election.cc:304] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.650131 1381 raft_consensus.cc:2747] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.650180 1273 consensus_queue.cc:1035] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:17.650956 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet f8d3cbdda0c94e139259d90919dc2f95 (table default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
W20250905 19:52:17.651916 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af5748bea3bd4f51a0d6c28a1d9044a0 consensus state has no leader
I20250905 19:52:17.652127 1185 raft_consensus.cc:2241] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.652279 1185 pending_rounds.cc:70] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:17.652407 1185 pending_rounds.cc:77] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089531262226432 op_type: NO_OP noop_request { }
I20250905 19:52:17.652508 1185 raft_consensus.cc:2887] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:17.652736 1185 raft_consensus.cc:2270] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.653440 1381 raft_consensus.cc:491] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.653569 1381 raft_consensus.cc:513] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.654211 1381 leader_election.cc:290] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.654708 1203 raft_consensus.cc:491] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.654860 1203 raft_consensus.cc:513] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.654937 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a884eca0e54a44dc920114e61a199689" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.655403 984 leader_election.cc:343] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.655609 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.655771 1203 leader_election.cc:290] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:17.656577 711 leader_election.cc:343] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6f4ec474911941b2a2d0825cdb37bd72
I20250905 19:52:17.657090 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b (DEFAULT_TABLE table=default.Table_ab2b806012994ad38e6f7a3171f00777 [id=29f567c56184458c96d24eb78f402137]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.657537 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ae83eea5f74e9187713e8b82c9a23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.659044 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.661159 1185 ts_tablet_manager.cc:1905] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:17.661729 712 leader_election.cc:343] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 6f4ec474911941b2a2d0825cdb37bd72
I20250905 19:52:17.661931 712 leader_election.cc:304] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:17.662314 1203 raft_consensus.cc:2747] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.663220 1186 tablet_replica.cc:331] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:17.663448 1186 raft_consensus.cc:2241] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.663635 1186 raft_consensus.cc:2270] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.663826 748 tablet_service.cc:1468] Processing CreateTablet for tablet 0267ac3e3f1a45978807737eca5d91ef (DEFAULT_TABLE table=default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.664234 1186 ts_tablet_manager.cc:1905] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.664304 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267ac3e3f1a45978807737eca5d91ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.666170 747 tablet_service.cc:1468] Processing CreateTablet for tablet 05ec559d0f83445892f8285f9bdeb2c8 (DEFAULT_TABLE table=default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.666589 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ec559d0f83445892f8285f9bdeb2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.665273 744 tablet_service.cc:1468] Processing CreateTablet for tablet 00d8144bb4f14358af539b7aae326918 (DEFAULT_TABLE table=default.Table_9ae350716fe144629f1f50ebee119d72 [id=92a9305bd65247c7912a68913af4e967]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.668680 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d8144bb4f14358af539b7aae326918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.675335 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d7d7567807594201a598d715ffbb318b (table default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
W20250905 19:52:17.676080 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 544dffc8b77049c295921d0af091b17d consensus state has no leader
I20250905 19:52:17.676963 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b4ad86cd8af24e86b952d94a60b5fa41" } new_table_name: "default.Table_68b569f22b1245008a2bd45d0be4d8cf" modify_external_catalogs: true
I20250905 19:52:17.677022 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 (DEFAULT_TABLE table=default.Table_6a7e530e3936440fa898546212620a3b [id=cf56ce3a388047dc9bd2d846817fef73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.676636 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 00d8144bb4f14358af539b7aae326918 (DEFAULT_TABLE table=default.Table_9ae350716fe144629f1f50ebee119d72 [id=92a9305bd65247c7912a68913af4e967]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.677423 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42f3c1efb94e446fb0128b6b412821e2. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.677629 986 consensus_peers.cc:489] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:17.679037 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.679939 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a51d8a846bff490aa865970a71596f5e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.680510 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a51d8a846bff490aa865970a71596f5e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.682058 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0aa187e069454d9abe28d7400d6067e7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.682562 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0aa187e069454d9abe28d7400d6067e7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.683362 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 0267ac3e3f1a45978807737eca5d91ef (DEFAULT_TABLE table=default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.683802 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267ac3e3f1a45978807737eca5d91ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.686437 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d8144bb4f14358af539b7aae326918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.680711 886 tablet_service.cc:1468] Processing CreateTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b (DEFAULT_TABLE table=default.Table_ab2b806012994ad38e6f7a3171f00777 [id=29f567c56184458c96d24eb78f402137]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.688271 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 05ec559d0f83445892f8285f9bdeb2c8 (DEFAULT_TABLE table=default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.688259 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_56e29cf512764408b243635ae01a6ab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:17.688573 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:17.688874 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56e29cf512764408b243635ae01a6ab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.688925 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ec559d0f83445892f8285f9bdeb2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.690166 891 tablet_service.cc:1468] Processing CreateTablet for tablet a5f076d9917f40d899d7422ad974fe0f (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.690630 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a884eca0e54a44dc920114e61a199689" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.690542 881 tablet_service.cc:1468] Processing CreateTablet for tablet 0267ac3e3f1a45978807737eca5d91ef (DEFAULT_TABLE table=default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.691748 887 tablet_service.cc:1468] Processing CreateTablet for tablet 6e89d23c4cb64f568b25f92a278b9d5e (DEFAULT_TABLE table=default.Table_b74cc30aa834421da92e129c76067622 [id=853047bedbb242d893b5be8b5a5c156e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:17.692186 986 leader_election.cc:343] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.692327 986 leader_election.cc:304] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.692621 1381 raft_consensus.cc:2747] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.692925 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6d09371944644b3082268b2b37ce54f8 (table default.Table_1eaa6c2091ee4d949f145d0e9e806550 [id=43eb792904d2448da31df6a56c647397]) successfully deleted
I20250905 19:52:17.691059 879 tablet_service.cc:1468] Processing CreateTablet for tablet 05ec559d0f83445892f8285f9bdeb2c8 (DEFAULT_TABLE table=default.Table_409df36c5b6f4d1aa96720b58745e361 [id=f5891226b675435f81cbdca3c8b4f43e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.697865 1028 tablet_service.cc:1468] Processing CreateTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 (DEFAULT_TABLE table=default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.702416 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.702638 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28e403dc2f6471ebea1805fd843cfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.714443 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_00ad894476894e1782714d41f84589b3" } new_table_name: "default.Table_f6b3a5016b7045bdbd914ad7ee173c25" modify_external_catalogs: true
I20250905 19:52:17.718644 882 tablet_service.cc:1468] Processing CreateTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 (DEFAULT_TABLE table=default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.719429 880 tablet_service.cc:1468] Processing CreateTablet for tablet 00d8144bb4f14358af539b7aae326918 (DEFAULT_TABLE table=default.Table_9ae350716fe144629f1f50ebee119d72 [id=92a9305bd65247c7912a68913af4e967]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.720021 878 tablet_service.cc:1468] Processing CreateTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 (DEFAULT_TABLE table=default.Table_6a7e530e3936440fa898546212620a3b [id=cf56ce3a388047dc9bd2d846817fef73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.720669 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ae83eea5f74e9187713e8b82c9a23b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.721123 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d8144bb4f14358af539b7aae326918. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.721355 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5f076d9917f40d899d7422ad974fe0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.721644 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42f3c1efb94e446fb0128b6b412821e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.722679 448 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]). Expected version 1 got 0
I20250905 19:52:17.728222 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f28e403dc2f6471ebea1805fd843cfb1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.734170 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05ec559d0f83445892f8285f9bdeb2c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.734251 914 raft_consensus.cc:1273] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Refusing update from remote peer 4eb3af79b01d4835bf6bf7b26bf016f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:17.736410 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a659dd4a6bcf4f618d1ddec59f33ffca" } modify_external_catalogs: true
I20250905 19:52:17.736651 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a659dd4a6bcf4f618d1ddec59f33ffca" } modify_external_catalogs: true
I20250905 19:52:17.740015 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0267ac3e3f1a45978807737eca5d91ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.740739 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f732ae0627c4da5b1691cde9b4d5562 consensus state has no leader
I20250905 19:52:17.745851 1169 tablet_bootstrap.cc:492] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.746300 1169 ts_tablet_manager.cc:1397] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.187s user 0.001s sys 0.008s
I20250905 19:52:17.747040 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a51d8a846bff490aa865970a71596f5e" } modify_external_catalogs: true
I20250905 19:52:17.747177 1169 raft_consensus.cc:357] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.747244 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a51d8a846bff490aa865970a71596f5e" } modify_external_catalogs: true
I20250905 19:52:17.747383 1169 raft_consensus.cc:383] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.747471 1169 raft_consensus.cc:738] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.747767 1169 consensus_queue.cc:260] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.748347 1169 ts_tablet_manager.cc:1428] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250905 19:52:17.752122 1394 raft_consensus.cc:491] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.752367 1394 raft_consensus.cc:513] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.753255 1394 leader_election.cc:290] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.758409 453 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet bfd550953f45423385e372222c69ee20
I20250905 19:52:17.760839 453 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]). Expected version 1 got 0
I20250905 19:52:17.760972 1169 tablet_bootstrap.cc:492] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.761142 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c8a8848304491bbd4a7971f96322f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:17.762024 986 leader_election.cc:343] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.762543 1236 consensus_queue.cc:1035] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:17.763128 1169 tablet_bootstrap.cc:654] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.765750 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e89d23c4cb64f568b25f92a278b9d5e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.767683 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_32577b5aebfd4f37b4400d62e431d7f1" } modify_external_catalogs: true
I20250905 19:52:17.767997 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_32577b5aebfd4f37b4400d62e431d7f1" } modify_external_catalogs: true
I20250905 19:52:17.776880 1184 ts_tablet_manager.cc:1918] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.777408 1184 log.cc:1199] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d7d7567807594201a598d715ffbb318b
I20250905 19:52:17.778383 1236 raft_consensus.cc:491] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.778553 1236 raft_consensus.cc:513] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.779034 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f with delete_type TABLET_DATA_DELETED (replaced by tablet 544dffc8b77049c295921d0af091b17d) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.781751 1171 tablet_bootstrap.cc:492] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:17.781966 1236 leader_election.cc:290] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.782410 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_315b3dd493ac42fea5c100b3e58d9533" } modify_external_catalogs: true
I20250905 19:52:17.782599 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_315b3dd493ac42fea5c100b3e58d9533" } modify_external_catalogs: true
I20250905 19:52:17.783335 1411 tablet.cc:1722] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:17.784145 1169 tablet_bootstrap.cc:492] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.784358 1169 ts_tablet_manager.cc:1397] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.014s
W20250905 19:52:17.784965 984 consensus_peers.cc:489] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:17.785190 1169 raft_consensus.cc:357] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.785399 1169 raft_consensus.cc:383] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.785480 1169 raft_consensus.cc:738] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.785740 1169 consensus_queue.cc:260] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.788152 1186 ts_tablet_manager.cc:1918] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.789180 1169 ts_tablet_manager.cc:1428] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:17.789283 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c8a8848304491bbd4a7971f96322f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:17.789829 1169 tablet_bootstrap.cc:492] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.790409 1184 ts_tablet_manager.cc:1939] T d7d7567807594201a598d715ffbb318b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:17.792939 1184 tablet_replica.cc:331] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:17.793131 1184 raft_consensus.cc:2241] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.793282 1184 raft_consensus.cc:2270] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.794054 1186 log.cc:1199] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6f4ec474911941b2a2d0825cdb37bd72
I20250905 19:52:17.794777 1186 ts_tablet_manager.cc:1939] T 6f4ec474911941b2a2d0825cdb37bd72 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:17.794879 1184 ts_tablet_manager.cc:1905] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.795519 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d7d7567807594201a598d715ffbb318b (table default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
W20250905 19:52:17.795786 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 0edac849e8834894be2d4ff323edd13d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0edac849e8834894be2d4ff323edd13d already in progress: creating tablet
I20250905 19:52:17.796016 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6f4ec474911941b2a2d0825cdb37bd72 (table default.Table_7dd83ce427b9438990eee889da0bcb0f [id=8904fcfbf1a94032b62d596e1ea14569]) successfully deleted
I20250905 19:52:17.796277 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:17.796327 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a8a32f742eaf473e9f6f9bf214f53679 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8a32f742eaf473e9f6f9bf214f53679 already in progress: creating tablet
I20250905 19:52:17.801172 1186 tablet_replica.cc:331] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:17.801388 1186 raft_consensus.cc:2241] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.802186 1186 raft_consensus.cc:2270] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.802896 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ae86cb11e24d348e2f12e982f3ab9f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.803838 984 leader_election.cc:343] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.804015 984 leader_election.cc:304] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:17.804463 984 leader_election.cc:343] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.804682 1236 raft_consensus.cc:2747] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.804878 1381 raft_consensus.cc:491] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.805011 1381 raft_consensus.cc:513] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.805441 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0039b8f0ba704907b2820c9e3a69cd73. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.805588 1381 leader_election.cc:290] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.807947 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 53ac46ce05694bc2b53e195638be9e23
I20250905 19:52:17.808188 1171 ts_tablet_manager.cc:1397] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.294s user 0.009s sys 0.005s
I20250905 19:52:17.809952 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.815001 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7f5bdee0d6b34929b57b46c6055c4ad6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:17.815111 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 4edfa9a656554e02b6649e8430c215fb (DEFAULT_TABLE table=default.Table_a19dd25caac94580b689031eda61e7e4 [id=859765611d234a4ba15dd451416fd2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.815515 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:17.815594 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4edfa9a656554e02b6649e8430c215fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.816202 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4d2ed1bc09c94526bc9662f2d4d21f8f
W20250905 19:52:17.815542 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.820981 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.827107 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3ae86cb11e24d348e2f12e982f3ab9f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.811929 1273 raft_consensus.cc:491] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.831418 1273 raft_consensus.cc:513] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.831787 447 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_68b569f22b1245008a2bd45d0be4d8cf [id=152edbecfe3648659c48a2cf89ff9798] alter complete (version 1)
I20250905 19:52:17.833084 1186 ts_tablet_manager.cc:1905] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.833511 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:17.833961 986 leader_election.cc:343] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.834565 986 leader_election.cc:304] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.835806 1170 tablet_bootstrap.cc:492] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:17.835997 1170 ts_tablet_manager.cc:1397] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.335s user 0.006s sys 0.026s
I20250905 19:52:17.836472 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.838029 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 2683a36998674f739b41d2a57339954c (DEFAULT_TABLE table=default.Table_97197831cd2f45cc909b9d7ab0caebc2 [id=9c5a9ace09234365a465b92fd6bb7f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.838187 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 53ac46ce05694bc2b53e195638be9e23 (DEFAULT_TABLE table=default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.839320 1025 tablet_service.cc:1468] Processing CreateTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb (DEFAULT_TABLE table=default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.839479 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 61667a379974487195f81043683b9a24 (DEFAULT_TABLE table=default.Table_4bdeb93947c346b1a36458d0dd5f5d2a [id=26cf482b4d50422eb27db543a7723f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.839771 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 4a518797d26140f5b9abe41eab415040 (DEFAULT_TABLE table=default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.839946 883 tablet_service.cc:1468] Processing CreateTablet for tablet 9b3acf17eab64913b5b33677196a5781 (DEFAULT_TABLE table=default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.840061 1394 raft_consensus.cc:2747] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.840068 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.840516 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d6daaa0fe43414299c032d1f4d9bb6f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:17.840916 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.841111 749 tablet_service.cc:1468] Processing CreateTablet for tablet ff2f289c293f41379e4af470d747860c (DEFAULT_TABLE table=default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.842731 886 tablet_service.cc:1468] Processing CreateTablet for tablet 4edfa9a656554e02b6649e8430c215fb (DEFAULT_TABLE table=default.Table_a19dd25caac94580b689031eda61e7e4 [id=859765611d234a4ba15dd451416fd2e9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.843065 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2683a36998674f739b41d2a57339954c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.842051 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d6daaa0fe43414299c032d1f4d9bb6f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:17.843505 753 tablet_service.cc:1468] Processing CreateTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 (DEFAULT_TABLE table=default.Table_6a7e530e3936440fa898546212620a3b [id=cf56ce3a388047dc9bd2d846817fef73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.843801 1027 tablet_service.cc:1468] Processing CreateTablet for tablet ff2f289c293f41379e4af470d747860c (DEFAULT_TABLE table=default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.844174 893 tablet_service.cc:1468] Processing CreateTablet for tablet ff2f289c293f41379e4af470d747860c (DEFAULT_TABLE table=default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:17.844578 984 leader_election.cc:343] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.845156 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 9b3acf17eab64913b5b33677196a5781 (DEFAULT_TABLE table=default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.845239 748 tablet_service.cc:1468] Processing CreateTablet for tablet 61667a379974487195f81043683b9a24 (DEFAULT_TABLE table=default.Table_4bdeb93947c346b1a36458d0dd5f5d2a [id=26cf482b4d50422eb27db543a7723f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.845537 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:17.845798 1053 raft_consensus.cc:2466] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:17.846477 986 leader_election.cc:343] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.846714 849 leader_election.cc:304] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:17.846755 986 leader_election.cc:304] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.847983 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9ae350716fe144629f1f50ebee119d72" } modify_external_catalogs: true
I20250905 19:52:17.848034 1394 raft_consensus.cc:2747] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.848191 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9ae350716fe144629f1f50ebee119d72" } modify_external_catalogs: true
I20250905 19:52:17.852865 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a2268b9408854caaac5ebd773623563b" } new_table_name: "default.Table_be613708cf304dc2856d98a4e1f6920f" modify_external_catalogs: true
I20250905 19:52:17.854480 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 00d8144bb4f14358af539b7aae326918
I20250905 19:52:17.854593 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.855228 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_409df36c5b6f4d1aa96720b58745e361" } new_table_name: "default.Table_9fe38a14d69846b8b468fdcef3fff0f5" modify_external_catalogs: true
I20250905 19:52:17.855898 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:17.857342 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61667a379974487195f81043683b9a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.860124 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a518797d26140f5b9abe41eab415040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.862284 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b12931248d47450dad62ed65df3edf73"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.862881 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b12931248d47450dad62ed65df3edf73 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.864197 1394 raft_consensus.cc:491] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.864303 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ac46ce05694bc2b53e195638be9e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.864365 1394 raft_consensus.cc:513] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.865101 1394 leader_election.cc:290] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.868103 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff2f289c293f41379e4af470d747860c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.870095 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_264ad09719304e68b1ef82f9ec5e5c22"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:17.870676 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_264ad09719304e68b1ef82f9ec5e5c22 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:17.871923 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13d282e1dc54d1faa2882c9b39d17cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.861920 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:17.874974 747 tablet_service.cc:1468] Processing CreateTablet for tablet 4edfa9a656554e02b6649e8430c215fb (DEFAULT_TABLE table=default.Table_a19dd25caac94580b689031eda61e7e4 [id=859765611d234a4ba15dd451416fd2e9]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:17.876055 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 68f2ccd8a7a64b89b44f2d6767ec3466 consensus state has no leader
I20250905 19:52:17.878394 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.880064 1394 raft_consensus.cc:491] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.880264 1394 raft_consensus.cc:513] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.880990 745 tablet_service.cc:1468] Processing CreateTablet for tablet 2683a36998674f739b41d2a57339954c (DEFAULT_TABLE table=default.Table_97197831cd2f45cc909b9d7ab0caebc2 [id=9c5a9ace09234365a465b92fd6bb7f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.881078 1394 leader_election.cc:290] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:17.881232 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b3acf17eab64913b5b33677196a5781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.882673 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a5c85dbda140d882a5a63a13aaabd7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:17.882987 848 leader_election.cc:343] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.883144 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.883287 1171 raft_consensus.cc:357] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.883455 1171 raft_consensus.cc:383] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.884769 755 tablet_service.cc:1468] Processing CreateTablet for tablet 53ac46ce05694bc2b53e195638be9e23 (DEFAULT_TABLE table=default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.885429 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff2f289c293f41379e4af470d747860c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.886070 1171 raft_consensus.cc:738] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.886351 1171 consensus_queue.cc:260] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.887450 986 leader_election.cc:343] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.888024 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1664cbe248514535953a1ef2d312f98f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:17.888746 986 leader_election.cc:343] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.895998 752 tablet_service.cc:1468] Processing CreateTablet for tablet 9b3acf17eab64913b5b33677196a5781 (DEFAULT_TABLE table=default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.896070 756 tablet_service.cc:1468] Processing CreateTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb (DEFAULT_TABLE table=default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.896486 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2683a36998674f739b41d2a57339954c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.897365 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a5c85dbda140d882a5a63a13aaabd7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.898031 984 leader_election.cc:343] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.898207 984 leader_election.cc:304] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.898934 1394 raft_consensus.cc:2747] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.899804 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b3acf17eab64913b5b33677196a5781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.901018 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13d282e1dc54d1faa2882c9b39d17cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.902186 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:17.902324 1169 tablet_bootstrap.cc:654] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.902645 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1664cbe248514535953a1ef2d312f98f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.903265 984 leader_election.cc:343] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.903437 984 leader_election.cc:304] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:17.907233 1413 raft_consensus.cc:2802] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:17.907357 1413 raft_consensus.cc:491] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:17.907460 1413 raft_consensus.cc:3058] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:17.908983 1394 raft_consensus.cc:2747] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:17.910030 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4edfa9a656554e02b6649e8430c215fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.910135 880 tablet_service.cc:1468] Processing CreateTablet for tablet 4a518797d26140f5b9abe41eab415040 (DEFAULT_TABLE table=default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.910816 1413 raft_consensus.cc:513] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.911047 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:17.911302 890 tablet_service.cc:1468] Processing CreateTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb (DEFAULT_TABLE table=default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.909794 1170 raft_consensus.cc:357] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.911547 1170 raft_consensus.cc:383] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.912012 1413 leader_election.cc:290] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:17.912048 1170 raft_consensus.cc:738] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.912084 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ac46ce05694bc2b53e195638be9e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.912346 1170 consensus_queue.cc:260] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.914350 1170 ts_tablet_manager.cc:1428] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.077s user 0.000s sys 0.002s
I20250905 19:52:17.914683 1170 tablet_bootstrap.cc:492] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:17.880016 744 tablet_service.cc:1468] Processing CreateTablet for tablet 4a518797d26140f5b9abe41eab415040 (DEFAULT_TABLE table=default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:17.916316 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05ec559d0f83445892f8285f9bdeb2c8 consensus state has no leader
I20250905 19:52:17.920478 1273 leader_election.cc:290] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:17.920398 878 tablet_service.cc:1468] Processing CreateTablet for tablet 61667a379974487195f81043683b9a24 (DEFAULT_TABLE table=default.Table_4bdeb93947c346b1a36458d0dd5f5d2a [id=26cf482b4d50422eb27db543a7723f2f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.920428 888 tablet_service.cc:1468] Processing CreateTablet for tablet 2683a36998674f739b41d2a57339954c (DEFAULT_TABLE table=default.Table_97197831cd2f45cc909b9d7ab0caebc2 [id=9c5a9ace09234365a465b92fd6bb7f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.920857 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4edfa9a656554e02b6649e8430c215fb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.921247 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a518797d26140f5b9abe41eab415040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.921675 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2683a36998674f739b41d2a57339954c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.921872 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b13d282e1dc54d1faa2882c9b39d17cb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.925084 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61667a379974487195f81043683b9a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.925017 889 tablet_service.cc:1468] Processing CreateTablet for tablet 53ac46ce05694bc2b53e195638be9e23 (DEFAULT_TABLE table=default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.926241 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61667a379974487195f81043683b9a24. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.931181 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 53ac46ce05694bc2b53e195638be9e23. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.935308 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9b3acf17eab64913b5b33677196a5781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.935751 1184 ts_tablet_manager.cc:1918] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:17.936242 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a518797d26140f5b9abe41eab415040. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.936460 1184 log.cc:1199] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/59faa6716fed4ea6b563df5f024b2058
I20250905 19:52:17.935875 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:17.937006 1184 ts_tablet_manager.cc:1939] T 59faa6716fed4ea6b563df5f024b2058 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:17.938436 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b776063b81c44217b517a619af927c08 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:17.942698 848 leader_election.cc:343] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.942987 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff2f289c293f41379e4af470d747860c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.944773 888 tablet_service.cc:1468] Processing CreateTablet for tablet baefaa1f042348c1874e306f1e5a8781 (DEFAULT_TABLE table=default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.945421 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baefaa1f042348c1874e306f1e5a8781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.945709 1184 tablet_replica.cc:331] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:17.946030 1184 raft_consensus.cc:2241] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:17.946327 1184 raft_consensus.cc:2270] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:17.948345 1169 tablet_bootstrap.cc:492] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.948666 1169 ts_tablet_manager.cc:1397] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.159s user 0.001s sys 0.006s
I20250905 19:52:17.950001 1184 ts_tablet_manager.cc:1905] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:17.950457 885 tablet_service.cc:1468] Processing CreateTablet for tablet 723168b075874dfcb3eb50c5134c499e (DEFAULT_TABLE table=default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.950459 1169 raft_consensus.cc:357] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.951251 1169 raft_consensus.cc:383] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.951526 1169 raft_consensus.cc:738] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.951716 1273 raft_consensus.cc:491] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.952579 1169 consensus_queue.cc:260] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.959924 1413 raft_consensus.cc:491] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.960660 751 tablet_service.cc:1468] Processing CreateTablet for tablet baefaa1f042348c1874e306f1e5a8781 (DEFAULT_TABLE table=default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.961117 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 42f3c1efb94e446fb0128b6b412821e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.961549 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baefaa1f042348c1874e306f1e5a8781. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.961467 1262 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20250905 19:52:17.961707 1273 raft_consensus.cc:513] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:17.961889 1262 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:17.962025 1262 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250905 19:52:17.962419 1273 leader_election.cc:290] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:17.963148 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:17.963706 848 leader_election.cc:343] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:17.963873 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723168b075874dfcb3eb50c5134c499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.970959 1169 ts_tablet_manager.cc:1428] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.022s user 0.000s sys 0.003s
I20250905 19:52:17.971724 1413 raft_consensus.cc:513] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.972424 1169 tablet_bootstrap.cc:492] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:17.973062 1413 leader_election.cc:290] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:17.974285 1169 tablet_bootstrap.cc:654] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:17.980734 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c6295c8c7874df89662b8cef6459fec" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:17.982362 882 tablet_service.cc:1468] Processing CreateTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 (DEFAULT_TABLE table=default.Table_3fa477c60f534a63b2cd4e54fe7c5c40 [id=8ce99397ad9f43508a20590a0d175ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.982633 891 tablet_service.cc:1468] Processing CreateTablet for tablet 415508e72ee34bc391c4369b5f8a2f48 (DEFAULT_TABLE table=default.Table_834cb637abd74d7694ba9e3f8e4ed84a [id=92aa73bcd0ad43d69787ac8ed44d6a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.982645 750 tablet_service.cc:1468] Processing CreateTablet for tablet 723168b075874dfcb3eb50c5134c499e (DEFAULT_TABLE table=default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.982836 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ac56c449ba4b7b8dbd5e87120799d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.983011 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415508e72ee34bc391c4369b5f8a2f48. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:17.983297 848 leader_election.cc:343] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:17.985683 1262 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 19:52:17.986173 1262 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:17.986330 1262 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 19:52:17.986459 1262 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 19:52:17.988312 887 tablet_service.cc:1468] Processing CreateTablet for tablet ac2e51a9071c4e1dae9e147687291175 (DEFAULT_TABLE table=default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.988736 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2e51a9071c4e1dae9e147687291175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:17.992231 1169 tablet_bootstrap.cc:492] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:17.992400 1169 ts_tablet_manager.cc:1397] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.020s user 0.001s sys 0.002s
I20250905 19:52:17.993254 1169 raft_consensus.cc:357] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.993474 1169 raft_consensus.cc:383] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:17.993566 1169 raft_consensus.cc:738] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:17.993808 1169 consensus_queue.cc:260] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:17.996930 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9e9994863c374354838aa0f817d0057e (DEFAULT_TABLE table=default.Table_56e29cf512764408b243635ae01a6ab0 [id=aad015c74608490985ee75fbd380f0e3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.997673 1169 ts_tablet_manager.cc:1428] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:17.998940 879 tablet_service.cc:1468] Processing CreateTablet for tablet 9630f1feb6d74472b5f6f4b54151311c (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:17.999022 1413 raft_consensus.cc:491] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:17.999318 1413 raft_consensus.cc:513] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.000010 1413 leader_election.cc:290] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.002863 1169 tablet_bootstrap.cc:492] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.003427 746 tablet_service.cc:1468] Processing CreateTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d (DEFAULT_TABLE table=default.Table_b12931248d47450dad62ed65df3edf73 [id=a5ecc77067244a09b1b8b3cea3ed44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.003188 1184 ts_tablet_manager.cc:1918] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.004446 1184 log.cc:1199] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/76a5c85dbda140d882a5a63a13aaabd7
I20250905 19:52:18.004534 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e9994863c374354838aa0f817d0057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.005232 1262 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=44671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:18.006318 1184 ts_tablet_manager.cc:1939] T 76a5c85dbda140d882a5a63a13aaabd7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.007920 1169 tablet_bootstrap.cc:654] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.007920 1262 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:18.008137 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9630f1feb6d74472b5f6f4b54151311c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.008589 1170 tablet_bootstrap.cc:654] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.008637 1184 tablet_replica.cc:331] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.008881 1184 raft_consensus.cc:2241] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.009052 1184 raft_consensus.cc:2270] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.009383 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be73873da994402e84fff6aa2ded7525" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.010174 848 leader_election.cc:343] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.010390 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723168b075874dfcb3eb50c5134c499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.010637 1184 ts_tablet_manager.cc:1905] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.010826 1262 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:52:18.019589 1171 ts_tablet_manager.cc:1428] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.208s user 0.007s sys 0.003s
I20250905 19:52:18.022466 1184 ts_tablet_manager.cc:1918] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.022648 1184 log.cc:1199] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/09b46a12076e492e89d0c0178bcf0524
I20250905 19:52:18.023051 1184 ts_tablet_manager.cc:1939] T 09b46a12076e492e89d0c0178bcf0524 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.024626 878 tablet_service.cc:1468] Processing CreateTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d (DEFAULT_TABLE table=default.Table_b12931248d47450dad62ed65df3edf73 [id=a5ecc77067244a09b1b8b3cea3ed44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.025179 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a73e984cbb4268b563f1ca01f4c65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.026536 1169 tablet_bootstrap.cc:492] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.026721 1169 ts_tablet_manager.cc:1397] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.024s user 0.001s sys 0.004s
I20250905 19:52:18.027215 1171 tablet_bootstrap.cc:492] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.027529 1169 raft_consensus.cc:357] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.027746 1169 raft_consensus.cc:383] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.027833 1169 raft_consensus.cc:738] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.028553 1169 consensus_queue.cc:260] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.008211 743 tablet_service.cc:1468] Processing CreateTablet for tablet a5af8232d31240ff855b79dcea29396b (DEFAULT_TABLE table=default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.029469 1171 tablet_bootstrap.cc:654] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.029932 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5af8232d31240ff855b79dcea29396b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.009791 742 tablet_service.cc:1468] Processing CreateTablet for tablet 9e9994863c374354838aa0f817d0057e (DEFAULT_TABLE table=default.Table_56e29cf512764408b243635ae01a6ab0 [id=aad015c74608490985ee75fbd380f0e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:18.031850 1427 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:18.032088 1169 ts_tablet_manager.cc:1428] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250905 19:52:18.033201 889 tablet_service.cc:1468] Processing CreateTablet for tablet a5af8232d31240ff855b79dcea29396b (DEFAULT_TABLE table=default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.033594 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5af8232d31240ff855b79dcea29396b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:18.034740 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 188043us
I20250905 19:52:18.035159 741 tablet_service.cc:1468] Processing CreateTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 (DEFAULT_TABLE table=default.Table_3fa477c60f534a63b2cd4e54fe7c5c40 [id=8ce99397ad9f43508a20590a0d175ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.036003 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ac56c449ba4b7b8dbd5e87120799d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.036092 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.036254 1052 raft_consensus.cc:3058] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.037297 1169 tablet_bootstrap.cc:492] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.038347 1185 ts_tablet_manager.cc:1918] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:18.038475 1185 log.cc:1199] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/7aa06165136a4fa5a935dfdd786a75c4
I20250905 19:52:18.038955 1185 ts_tablet_manager.cc:1939] T 7aa06165136a4fa5a935dfdd786a75c4 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:18.039613 1169 tablet_bootstrap.cc:654] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.039708 1052 raft_consensus.cc:2466] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:18.040098 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 7aa06165136a4fa5a935dfdd786a75c4 (table default.Table_16b3928f208649fa879486e9d7d6a235 [id=ab02be958b034155ba9d2a431944a420]) successfully deleted
I20250905 19:52:18.040228 1411 tablet.cc:1722] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.040721 1184 tablet_replica.cc:331] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.040980 1184 raft_consensus.cc:2241] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.041177 1184 raft_consensus.cc:2270] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.040027 740 tablet_service.cc:1468] Processing CreateTablet for tablet 415508e72ee34bc391c4369b5f8a2f48 (DEFAULT_TABLE table=default.Table_834cb637abd74d7694ba9e3f8e4ed84a [id=92aa73bcd0ad43d69787ac8ed44d6a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.041832 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415508e72ee34bc391c4369b5f8a2f48. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.042120 1184 ts_tablet_manager.cc:1905] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.042186 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be73873da994402e84fff6aa2ded7525" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.042464 1051 raft_consensus.cc:2466] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:18.042781 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c6295c8c7874df89662b8cef6459fec" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.042948 1053 raft_consensus.cc:2466] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:18.044188 738 tablet_service.cc:1468] Processing CreateTablet for tablet 9630f1feb6d74472b5f6f4b54151311c (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.046051 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e9994863c374354838aa0f817d0057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.048830 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.049099 1052 raft_consensus.cc:2466] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:18.049484 739 tablet_service.cc:1468] Processing CreateTablet for tablet ac2e51a9071c4e1dae9e147687291175 (DEFAULT_TABLE table=default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.051021 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9630f1feb6d74472b5f6f4b54151311c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.051728 1184 ts_tablet_manager.cc:1918] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.051857 1184 log.cc:1199] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9d6daaa0fe43414299c032d1f4d9bb6f
I20250905 19:52:18.052263 1184 ts_tablet_manager.cc:1939] T 9d6daaa0fe43414299c032d1f4d9bb6f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.052755 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2e51a9071c4e1dae9e147687291175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.058223 849 leader_election.cc:304] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.058555 1408 raft_consensus.cc:2802] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.058997 1408 raft_consensus.cc:695] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:18.059365 1408 consensus_queue.cc:237] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.060565 1184 tablet_replica.cc:331] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.061282 1184 raft_consensus.cc:2241] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.061455 1184 raft_consensus.cc:2270] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.061677 1185 tablet_replica.cc:331] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:18.061940 1185 raft_consensus.cc:2241] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.062211 1185 raft_consensus.cc:2270] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.063577 1184 ts_tablet_manager.cc:1905] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.063717 1186 ts_tablet_manager.cc:1918] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.069137 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a73e984cbb4268b563f1ca01f4c65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.071377 1411 tablet.cc:1722] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:18.071938 1186 log.cc:1199] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/78ac5c7e9b6741d79db2e606d55a3509
I20250905 19:52:18.072484 1186 ts_tablet_manager.cc:1939] T 78ac5c7e9b6741d79db2e606d55a3509 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.073449 1186 tablet_replica.cc:331] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.073885 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 78ac5c7e9b6741d79db2e606d55a3509 (table default.Table_d6e9e58d90324cd2a7110d05c7588160 [id=5ac15645cb4d4730a4b4464a85299bed]) successfully deleted
I20250905 19:52:18.074251 1186 raft_consensus.cc:2241] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.074525 1186 raft_consensus.cc:2270] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.076067 1186 ts_tablet_manager.cc:1905] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.080847 1169 tablet_bootstrap.cc:492] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.080906 1185 ts_tablet_manager.cc:1905] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.081023 1169 ts_tablet_manager.cc:1397] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.044s user 0.001s sys 0.011s
I20250905 19:52:18.082001 1169 raft_consensus.cc:357] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.082187 1169 raft_consensus.cc:383] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.082283 1169 raft_consensus.cc:738] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.082599 1169 consensus_queue.cc:260] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.082703 849 leader_election.cc:304] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.084241 1169 ts_tablet_manager.cc:1428] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:18.084399 451 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 2 got 0
I20250905 19:52:18.086313 1169 tablet_bootstrap.cc:492] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.087908 1169 tablet_bootstrap.cc:654] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.085549 1424 tablet.cc:1722] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.090154 1408 raft_consensus.cc:2802] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
W20250905 19:52:18.090205 1428 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:18.092548 1184 ts_tablet_manager.cc:1918] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.092708 1184 log.cc:1199] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/1664cbe248514535953a1ef2d312f98f
I20250905 19:52:18.093575 1184 ts_tablet_manager.cc:1939] T 1664cbe248514535953a1ef2d312f98f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.094434 451 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]). Expected version 1 got 0
I20250905 19:52:18.094790 451 catalog_manager.cc:5582] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:18.096181 1184 tablet_replica.cc:331] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.096400 1184 raft_consensus.cc:2241] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:18.096827 1184 raft_consensus.cc:2270] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.096976 1408 raft_consensus.cc:491] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:18.097134 1408 raft_consensus.cc:3058] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.097729 1422 tablet.cc:1722] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.098294 1169 tablet_bootstrap.cc:492] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.098457 1169 ts_tablet_manager.cc:1397] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.001s
I20250905 19:52:18.099335 849 leader_election.cc:304] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.099393 1169 raft_consensus.cc:357] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.099571 1169 raft_consensus.cc:383] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.099649 1169 raft_consensus.cc:738] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.099922 1169 consensus_queue.cc:260] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.100692 849 leader_election.cc:304] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.101790 1169 ts_tablet_manager.cc:1428] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:18.100937 1417 raft_consensus.cc:2802] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:18.102949 1169 tablet_bootstrap.cc:492] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.105306 1169 tablet_bootstrap.cc:654] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:18.110210 1437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:18.111236 1419 raft_consensus.cc:2802] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:18.111415 1419 raft_consensus.cc:491] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:18.112044 1262 server_base.cc:1047] running on GCE node
I20250905 19:52:18.111554 1417 raft_consensus.cc:491] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:18.112430 1419 raft_consensus.cc:3058] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.113206 1417 raft_consensus.cc:3058] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.113432 1262 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20250905 19:52:18.117204 986 consensus_peers.cc:489] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:18.126098 1262 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:18.128244 1262 hybrid_clock.cc:648] HybridClock initialized: now 1757101938128203 us; error 55 us; skew 500 ppm
I20250905 19:52:18.128799 1262 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:18.134552 1419 raft_consensus.cc:513] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.136227 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:18.134445 1408 raft_consensus.cc:513] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:18.138772 848 leader_election.cc:343] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.139567 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.139808 1052 raft_consensus.cc:3058] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.140125 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be73873da994402e84fff6aa2ded7525" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:18.140758 848 leader_election.cc:343] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.142866 1052 raft_consensus.cc:2466] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:18.143851 849 leader_election.cc:304] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.144223 1413 raft_consensus.cc:2802] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.145117 1413 raft_consensus.cc:695] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:18.145099 1419 leader_election.cc:290] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.146399 1413 consensus_queue.cc:237] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.148643 1408 leader_election.cc:290] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.150471 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be73873da994402e84fff6aa2ded7525" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.150681 1052 raft_consensus.cc:3058] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.159739 1417 raft_consensus.cc:513] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.160337 1052 raft_consensus.cc:2466] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:18.161872 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c6295c8c7874df89662b8cef6459fec" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:18.163040 848 leader_election.cc:343] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.166281 1262 webserver.cc:480] Webserver started at http://127.0.104.60:44671/ using document root <none> and password file <none>
I20250905 19:52:18.166539 1185 ts_tablet_manager.cc:1918] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:18.166714 1185 log.cc:1199] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/17d0a48c25b84facb082670cb898468c
I20250905 19:52:18.167210 1185 ts_tablet_manager.cc:1939] T 17d0a48c25b84facb082670cb898468c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:18.167351 1262 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:18.167488 1262 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:18.168066 849 leader_election.cc:304] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.168395 1408 raft_consensus.cc:2802] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.168529 1408 raft_consensus.cc:695] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:18.168823 1408 consensus_queue.cc:237] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.169025 1417 leader_election.cc:290] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.171082 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1c6295c8c7874df89662b8cef6459fec" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.171360 1052 raft_consensus.cc:3058] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.171502 1170 tablet_bootstrap.cc:492] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.171674 1170 ts_tablet_manager.cc:1397] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.257s user 0.004s sys 0.002s
I20250905 19:52:18.172590 1424 tablet.cc:1722] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:18.173018 1170 raft_consensus.cc:357] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.173228 1170 raft_consensus.cc:383] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.173306 1170 raft_consensus.cc:738] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.173566 1170 consensus_queue.cc:260] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.174275 1170 ts_tablet_manager.cc:1428] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250905 19:52:18.174872 1052 raft_consensus.cc:2466] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:18.175083 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:34395, user_credentials={real_user=slave}} blocked reactor thread for 117186us
I20250905 19:52:18.175959 1026 tablet_service.cc:1468] Processing CreateTablet for tablet baefaa1f042348c1874e306f1e5a8781 (DEFAULT_TABLE table=default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.177731 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet baefaa1f042348c1874e306f1e5a8781. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.178296 1170 tablet_bootstrap.cc:492] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.180095 1170 tablet_bootstrap.cc:654] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.177377 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 8adc74a357244102801b2aa8d27d9747 with delete_type TABLET_DATA_DELETED (replaced by tablet 2201e4bad99643a9b041f6f99406433b) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.184499 1184 ts_tablet_manager.cc:1905] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.186003 1171 tablet_bootstrap.cc:492] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.186216 1171 ts_tablet_manager.cc:1397] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.159s user 0.001s sys 0.009s
I20250905 19:52:18.187171 849 leader_election.cc:304] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.187656 1417 raft_consensus.cc:2802] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.187920 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 723168b075874dfcb3eb50c5134c499e (DEFAULT_TABLE table=default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.188022 1417 raft_consensus.cc:695] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:18.188452 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 723168b075874dfcb3eb50c5134c499e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.188340 1417 consensus_queue.cc:237] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.191143 1184 ts_tablet_manager.cc:1918] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:18.191282 1184 log.cc:1199] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/7aa06165136a4fa5a935dfdd786a75c4
I20250905 19:52:18.191761 1184 ts_tablet_manager.cc:1939] T 7aa06165136a4fa5a935dfdd786a75c4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.192152 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d (DEFAULT_TABLE table=default.Table_b12931248d47450dad62ed65df3edf73 [id=a5ecc77067244a09b1b8b3cea3ed44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.192533 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89a73e984cbb4268b563f1ca01f4c65d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.194042 1186 ts_tablet_manager.cc:1918] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:18.194191 1186 log.cc:1199] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/17d0a48c25b84facb082670cb898468c
I20250905 19:52:18.194602 1186 ts_tablet_manager.cc:1939] T 17d0a48c25b84facb082670cb898468c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.195653 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]) successfully deleted
I20250905 19:52:18.198285 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]) successfully deleted
I20250905 19:52:18.199560 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a5af8232d31240ff855b79dcea29396b (DEFAULT_TABLE table=default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:18.199604 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 05ae83eea5f74e9187713e8b82c9a23b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05ae83eea5f74e9187713e8b82c9a23b already in progress: creating tablet
I20250905 19:52:18.200105 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a5af8232d31240ff855b79dcea29396b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.187147 1171 raft_consensus.cc:357] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.200415 1171 raft_consensus.cc:383] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.200516 1171 raft_consensus.cc:738] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.204164 1186 tablet_replica.cc:331] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.204370 1186 raft_consensus.cc:2241] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.204594 1186 raft_consensus.cc:2270] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.205139 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 9e9994863c374354838aa0f817d0057e (DEFAULT_TABLE table=default.Table_56e29cf512764408b243635ae01a6ab0 [id=aad015c74608490985ee75fbd380f0e3]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:18.205322 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet b0702063ec1c4f6fba58e6d4f3912212 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0702063ec1c4f6fba58e6d4f3912212 already in progress: creating tablet
I20250905 19:52:18.205533 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9e9994863c374354838aa0f817d0057e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.206331 1170 tablet_bootstrap.cc:492] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.206470 1170 ts_tablet_manager.cc:1397] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
W20250905 19:52:18.206580 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 74a7e7d9874e4d26bda3ab24a6e24adc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74a7e7d9874e4d26bda3ab24a6e24adc already in progress: creating tablet
W20250905 19:52:18.207262 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 4fe94d1254d04949aa25346d8e323c3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fe94d1254d04949aa25346d8e323c3b already in progress: creating tablet
I20250905 19:52:18.207237 1170 raft_consensus.cc:357] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.207418 1170 raft_consensus.cc:383] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.207500 1170 raft_consensus.cc:738] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.207748 1170 consensus_queue.cc:260] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:18.207924 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 53ac46ce05694bc2b53e195638be9e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ac46ce05694bc2b53e195638be9e23 already in progress: creating tablet
W20250905 19:52:18.208472 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 00d8144bb4f14358af539b7aae326918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00d8144bb4f14358af539b7aae326918 already in progress: creating tablet
W20250905 19:52:18.209080 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2ed1bc09c94526bc9662f2d4d21f8f already in progress: creating tablet
I20250905 19:52:18.209496 1171 consensus_queue.cc:260] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.209952 1170 ts_tablet_manager.cc:1428] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:18.211012 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d23d6df63c1a4e02a43bccf625e200bb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.211352 1170 tablet_bootstrap.cc:492] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.213153 1170 tablet_bootstrap.cc:654] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.215253 1186 ts_tablet_manager.cc:1905] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.218946 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 (DEFAULT_TABLE table=default.Table_3fa477c60f534a63b2cd4e54fe7c5c40 [id=8ce99397ad9f43508a20590a0d175ad1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.220991 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00ac56c449ba4b7b8dbd5e87120799d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.222119 1176 raft_consensus.cc:491] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.222343 1176 raft_consensus.cc:513] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.223142 1176 leader_election.cc:290] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.224298 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c5807c1132747e6af79d93e7496d500" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:18.224409 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c5807c1132747e6af79d93e7496d500" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:18.224907 986 leader_election.cc:343] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.225073 984 leader_election.cc:343] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.225234 984 leader_election.cc:304] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.225768 1176 raft_consensus.cc:2747] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.216419 1171 ts_tablet_manager.cc:1428] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.030s user 0.003s sys 0.000s
I20250905 19:52:18.226172 1184 tablet_replica.cc:331] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.226864 1184 raft_consensus.cc:2241] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
W20250905 19:52:18.227057 440 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_9fe38a14d69846b8b468fdcef3fff0f5 [id=f5891226b675435f81cbdca3c8b4f43e], current schema version=2) failed: Not found: failed to reset TS proxy: No leader found: Tablet 05ec559d0f83445892f8285f9bdeb2c8 consensus state has no leader
I20250905 19:52:18.227155 1171 tablet_bootstrap.cc:492] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.229557 1171 tablet_bootstrap.cc:654] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.231000 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 57f851a6eecc4a41806bdaa19cc5afab (table default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]). Expected version 1 got 0
I20250905 19:52:18.231751 912 raft_consensus.cc:1273] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Refusing update from remote peer 4eb3af79b01d4835bf6bf7b26bf016f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:18.234990 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_14d9939868624cd49dc6e2facdc4eb08"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:18.228317 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 415508e72ee34bc391c4369b5f8a2f48 (DEFAULT_TABLE table=default.Table_834cb637abd74d7694ba9e3f8e4ed84a [id=92aa73bcd0ad43d69787ac8ed44d6a07]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.235625 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 415508e72ee34bc391c4369b5f8a2f48. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:18.235697 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_14d9939868624cd49dc6e2facdc4eb08 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.237345 448 catalog_manager.cc:5582] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:18.237823 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]). Expected version 1 got 0
I20250905 19:52:18.228842 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ac2e51a9071c4e1dae9e147687291175 (DEFAULT_TABLE table=default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.229221 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 9630f1feb6d74472b5f6f4b54151311c (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.241614 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac2e51a9071c4e1dae9e147687291175. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.242725 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9630f1feb6d74472b5f6f4b54151311c. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:18.231034 984 consensus_peers.cc:489] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:18.232571 1176 consensus_queue.cc:1035] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:18.253175 1170 tablet_bootstrap.cc:492] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.253371 1170 ts_tablet_manager.cc:1397] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.042s user 0.003s sys 0.000s
I20250905 19:52:18.254168 1170 raft_consensus.cc:357] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.254374 1170 raft_consensus.cc:383] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.254446 1170 raft_consensus.cc:738] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.254684 1170 consensus_queue.cc:260] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.257853 1171 tablet_bootstrap.cc:492] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.258054 1171 ts_tablet_manager.cc:1397] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.031s user 0.004s sys 0.000s
I20250905 19:52:18.259023 1170 ts_tablet_manager.cc:1428] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:18.259117 1171 raft_consensus.cc:357] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.259366 1171 raft_consensus.cc:383] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.259475 1171 raft_consensus.cc:738] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.259801 1171 consensus_queue.cc:260] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.260850 1170 tablet_bootstrap.cc:492] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.261216 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d7d7567807594201a598d715ffbb318b with delete_type TABLET_DATA_DELETED (replaced by tablet 57f851a6eecc4a41806bdaa19cc5afab) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.261405 1171 ts_tablet_manager.cc:1428] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:18.262174 1171 tablet_bootstrap.cc:492] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.263298 1170 tablet_bootstrap.cc:654] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.264534 1171 tablet_bootstrap.cc:654] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.265903 1186 ts_tablet_manager.cc:1918] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:18.266031 1186 log.cc:1199] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/7aa06165136a4fa5a935dfdd786a75c4
I20250905 19:52:18.270501 1186 ts_tablet_manager.cc:1939] T 7aa06165136a4fa5a935dfdd786a75c4 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.271206 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4] alter complete (version 2)
I20250905 19:52:18.272408 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 7aa06165136a4fa5a935dfdd786a75c4 (table default.Table_16b3928f208649fa879486e9d7d6a235 [id=ab02be958b034155ba9d2a431944a420]) successfully deleted
I20250905 19:52:18.274354 1184 raft_consensus.cc:2270] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:18.275249 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet b0702063ec1c4f6fba58e6d4f3912212 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0702063ec1c4f6fba58e6d4f3912212 already in progress: creating tablet
I20250905 19:52:18.277287 1170 tablet_bootstrap.cc:492] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.277467 1170 ts_tablet_manager.cc:1397] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250905 19:52:18.278437 1170 raft_consensus.cc:357] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.278656 1170 raft_consensus.cc:383] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.278749 1170 raft_consensus.cc:738] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:18.279584 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 05ae83eea5f74e9187713e8b82c9a23b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05ae83eea5f74e9187713e8b82c9a23b already in progress: creating tablet
W20250905 19:52:18.280486 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 74a7e7d9874e4d26bda3ab24a6e24adc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74a7e7d9874e4d26bda3ab24a6e24adc already in progress: creating tablet
W20250905 19:52:18.281253 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 53ac46ce05694bc2b53e195638be9e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ac46ce05694bc2b53e195638be9e23 already in progress: creating tablet
W20250905 19:52:18.282115 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4fe94d1254d04949aa25346d8e323c3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fe94d1254d04949aa25346d8e323c3b already in progress: creating tablet
W20250905 19:52:18.282997 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2ed1bc09c94526bc9662f2d4d21f8f already in progress: creating tablet
I20250905 19:52:18.287006 1171 tablet_bootstrap.cc:492] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.287631 1171 ts_tablet_manager.cc:1397] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.006s sys 0.001s
I20250905 19:52:18.288443 746 tablet_service.cc:1468] Processing CreateTablet for tablet 6f3473785fff45d692bd50b1015ffe39 (DEFAULT_TABLE table=default.Table_14d9939868624cd49dc6e2facdc4eb08 [id=c4985b481dc94bd48f9623e4a300c498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.288446 886 tablet_service.cc:1468] Processing CreateTablet for tablet 6f3473785fff45d692bd50b1015ffe39 (DEFAULT_TABLE table=default.Table_14d9939868624cd49dc6e2facdc4eb08 [id=c4985b481dc94bd48f9623e4a300c498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.288872 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3473785fff45d692bd50b1015ffe39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.288970 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3473785fff45d692bd50b1015ffe39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.291376 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0be185ca49de49f09cc109dc19ec7063"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.291945 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0be185ca49de49f09cc109dc19ec7063 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.294291 1171 raft_consensus.cc:357] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.294642 1171 raft_consensus.cc:383] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.294776 1171 raft_consensus.cc:738] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.295112 1171 consensus_queue.cc:260] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.295665 1186 tablet_replica.cc:331] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:18.296664 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 00d8144bb4f14358af539b7aae326918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00d8144bb4f14358af539b7aae326918 already in progress: creating tablet
I20250905 19:52:18.296478 1171 ts_tablet_manager.cc:1428] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250905 19:52:18.297609 1186 raft_consensus.cc:2241] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.297978 1171 tablet_bootstrap.cc:492] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.297992 1186 raft_consensus.cc:2270] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.298887 1186 ts_tablet_manager.cc:1905] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.299698 1171 tablet_bootstrap.cc:654] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.299418 1170 consensus_queue.cc:260] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.303436 1170 ts_tablet_manager.cc:1428] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.026s user 0.002s sys 0.001s
I20250905 19:52:18.305171 1170 tablet_bootstrap.cc:492] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.305441 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 305a7758527d4d52a669b2f2c4ceedbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.308523 1170 tablet_bootstrap.cc:654] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.297477 1184 ts_tablet_manager.cc:1905] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.310595 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_4bdeb93947c346b1a36458d0dd5f5d2a" } new_table_name: "default.Table_deab3e4fa1104b67b031b82f945d2917" modify_external_catalogs: true
I20250905 19:52:18.312268 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 302053e1611b4e22a9a2e7b60ae68f27 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.312373 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 57f851a6eecc4a41806bdaa19cc5afab from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.312964 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e2b7277d0ccd4d0aa90b08e13f6eb572" } new_table_name: "default.Table_580d1de7c8164c1c8c1e3ba556bd5c94" modify_external_catalogs: true
I20250905 19:52:18.313048 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_906ecce79e3a438c82eb0f15671e1d07"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:18.313453 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e65ca281d8804277a83ab7bfc2ab9cca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.313633 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_906ecce79e3a438c82eb0f15671e1d07 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:18.313983 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e65ca281d8804277a83ab7bfc2ab9cca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.314415 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9f1863dcd87d4fb98b0f2e89677b5edf" } new_table_name: "default.Table_092735da5fbd4ffab7939962df253fa0" modify_external_catalogs: true
I20250905 19:52:18.315025 1169 tablet_bootstrap.cc:492] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.314981 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e304b44cbe8f4c77bae2af66f9b5920e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:18.315088 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_79a0ddd169324b64b69f90b3d65bfa11"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:18.315193 1169 ts_tablet_manager.cc:1397] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.212s user 0.067s sys 0.078s
W20250905 19:52:18.315459 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79a0ddd169324b64b69f90b3d65bfa11 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.315649 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_bac4d4565a874689b77c9fa8009f96b4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.315459 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e304b44cbe8f4c77bae2af66f9b5920e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:18.316164 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bac4d4565a874689b77c9fa8009f96b4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.317251 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 59faa6716fed4ea6b563df5f024b2058 (table default.Table_9fe38a14d69846b8b468fdcef3fff0f5 [id=f5891226b675435f81cbdca3c8b4f43e]) successfully deleted
I20250905 19:52:18.317468 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 76a5c85dbda140d882a5a63a13aaabd7 (table default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]) successfully deleted
I20250905 19:52:18.317625 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 09b46a12076e492e89d0c0178bcf0524 (table default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a [id=3b8967eb30ad46cdbf272136f23318f1]) successfully deleted
W20250905 19:52:18.317783 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a8a32f742eaf473e9f6f9bf214f53679 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a8a32f742eaf473e9f6f9bf214f53679 already in progress: creating tablet
I20250905 19:52:18.317938 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9d6daaa0fe43414299c032d1f4d9bb6f (table default.Table_363859b13b284790a71ced21040081e4 [id=3d169e77e70f4a059cdf41f2f3a70171]) successfully deleted
I20250905 19:52:18.318114 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 1664cbe248514535953a1ef2d312f98f (table default.Table_6a14b2be50f942d8bfd8c47247512879 [id=353aa71a6ee64c41aa94daadd629d845]) successfully deleted
I20250905 19:52:18.318459 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 7aa06165136a4fa5a935dfdd786a75c4 (table default.Table_16b3928f208649fa879486e9d7d6a235 [id=ab02be958b034155ba9d2a431944a420]) successfully deleted
I20250905 19:52:18.322983 1169 raft_consensus.cc:357] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.323271 1169 raft_consensus.cc:383] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.323537 1169 raft_consensus.cc:738] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.323886 1169 consensus_queue.cc:260] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.324908 1169 ts_tablet_manager.cc:1428] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250905 19:52:18.325771 1170 tablet_bootstrap.cc:492] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.329537 1411 tablet.cc:1722] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.330927 1441 tablet.cc:1722] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.331640 1170 ts_tablet_manager.cc:1397] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.023s user 0.004s sys 0.000s
I20250905 19:52:18.336058 1413 raft_consensus.cc:491] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.336212 1413 raft_consensus.cc:513] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.336850 1413 leader_election.cc:290] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.338095 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57259d8eb0d44d1cbac80047c7ad67b6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.338600 848 leader_election.cc:343] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.339344 849 leader_election.cc:343] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:18.339483 849 leader_election.cc:304] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.339751 1413 raft_consensus.cc:2747] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.347530 886 tablet_service.cc:1468] Processing CreateTablet for tablet 44769f1170354870ae2d34fdb67ffbae (DEFAULT_TABLE table=default.Table_0be185ca49de49f09cc109dc19ec7063 [id=3235ea3e87b249e884b027b840d4fd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.349231 1169 tablet_bootstrap.cc:492] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.349859 746 tablet_service.cc:1468] Processing CreateTablet for tablet 44769f1170354870ae2d34fdb67ffbae (DEFAULT_TABLE table=default.Table_0be185ca49de49f09cc109dc19ec7063 [id=3235ea3e87b249e884b027b840d4fd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.350289 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44769f1170354870ae2d34fdb67ffbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.350925 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44769f1170354870ae2d34fdb67ffbae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.351204 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 6f3473785fff45d692bd50b1015ffe39 (DEFAULT_TABLE table=default.Table_14d9939868624cd49dc6e2facdc4eb08 [id=c4985b481dc94bd48f9623e4a300c498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.351542 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f3473785fff45d692bd50b1015ffe39. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.354040 1169 tablet_bootstrap.cc:654] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:18.354861 848 consensus_peers.cc:489] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:18.353433 1186 ts_tablet_manager.cc:1918] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.355800 1186 log.cc:1199] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/b776063b81c44217b517a619af927c08
I20250905 19:52:18.354709 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 44769f1170354870ae2d34fdb67ffbae (DEFAULT_TABLE table=default.Table_0be185ca49de49f09cc109dc19ec7063 [id=3235ea3e87b249e884b027b840d4fd77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.356501 1170 raft_consensus.cc:357] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.356729 1170 raft_consensus.cc:383] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.356812 1170 raft_consensus.cc:738] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.357209 1184 ts_tablet_manager.cc:1918] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:18.358649 1171 tablet_bootstrap.cc:492] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.358592 1170 consensus_queue.cc:260] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.358791 1171 ts_tablet_manager.cc:1397] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.062s user 0.003s sys 0.009s
I20250905 19:52:18.359781 1171 raft_consensus.cc:357] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.360001 1171 raft_consensus.cc:383] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.360093 1171 raft_consensus.cc:738] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.360381 1171 consensus_queue.cc:260] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.361023 1170 ts_tablet_manager.cc:1428] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250905 19:52:18.361414 1171 ts_tablet_manager.cc:1428] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:18.362181 1171 tablet_bootstrap.cc:492] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.364112 1171 tablet_bootstrap.cc:654] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.364997 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 44769f1170354870ae2d34fdb67ffbae. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:18.368003 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9a81376d4c0b47a9817deae4d34ffb47 consensus state has no leader
W20250905 19:52:18.369093 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61667a379974487195f81043683b9a24 consensus state has no leader
I20250905 19:52:18.363782 1170 tablet_bootstrap.cc:492] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.363925 1186 ts_tablet_manager.cc:1939] T b776063b81c44217b517a619af927c08 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.371022 1186 tablet_replica.cc:331] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.371209 1186 raft_consensus.cc:2241] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.371388 1186 raft_consensus.cc:2270] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.373780 1184 log.cc:1199] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/17d0a48c25b84facb082670cb898468c
I20250905 19:52:18.373847 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet b776063b81c44217b517a619af927c08 (table default.Table_ce8e455023774d02ba896de13623c0ea [id=06e2f51841544340adc3a39ca41a9bea]) successfully deleted
I20250905 19:52:18.373879 1052 raft_consensus.cc:1273] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Refusing update from remote peer 835daf30e4324782848290be181ae78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:18.374238 1184 ts_tablet_manager.cc:1939] T 17d0a48c25b84facb082670cb898468c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.375418 1186 ts_tablet_manager.cc:1905] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.375340 1408 consensus_queue.cc:1035] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:18.376675 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 17d0a48c25b84facb082670cb898468c (table default.Table_fe362aa06bd64814853c72f1a41c76b3 [id=59ebabf8b07641e8bc87c8003786c9ca]) successfully deleted
I20250905 19:52:18.379280 1170 tablet_bootstrap.cc:654] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:18.380686 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 05ae83eea5f74e9187713e8b82c9a23b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 05ae83eea5f74e9187713e8b82c9a23b already in progress: creating tablet
W20250905 19:52:18.384898 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet b0702063ec1c4f6fba58e6d4f3912212 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b0702063ec1c4f6fba58e6d4f3912212 already in progress: creating tablet
W20250905 19:52:18.386654 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 74a7e7d9874e4d26bda3ab24a6e24adc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 74a7e7d9874e4d26bda3ab24a6e24adc already in progress: creating tablet
I20250905 19:52:18.388197 1184 tablet_replica.cc:331] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.388468 1184 raft_consensus.cc:2241] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.388698 1184 raft_consensus.cc:2270] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.389328 1184 ts_tablet_manager.cc:1905] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.395185 1262 fs_manager.cc:714] Time spent opening directory manager: real 0.058s user 0.003s sys 0.001s
I20250905 19:52:18.398144 1441 tablet.cc:1722] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.398867 1411 tablet.cc:1722] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:18.403664 1186 ts_tablet_manager.cc:1918] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.409082 1186 log.cc:1199] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8adc74a357244102801b2aa8d27d9747
I20250905 19:52:18.410005 1186 ts_tablet_manager.cc:1939] T 8adc74a357244102801b2aa8d27d9747 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.411684 1169 tablet_bootstrap.cc:492] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.411886 1169 ts_tablet_manager.cc:1397] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.063s user 0.006s sys 0.001s
I20250905 19:52:18.412334 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 3602094b96c14e66b997e8035fc53c40 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.413266 1169 raft_consensus.cc:357] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.413506 1169 raft_consensus.cc:383] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.413564 1169 raft_consensus.cc:738] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:18.413671 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3602094b96c14e66b997e8035fc53c40 consensus state has no leader
I20250905 19:52:18.413722 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8adc74a357244102801b2aa8d27d9747 (table default.Table_8fe7ea747ff749c389bdd49a8291bf19 [id=1298f01fa92e446c9265c13bb9406ba8]) successfully deleted
I20250905 19:52:18.413830 1169 consensus_queue.cc:260] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.414309 1186 tablet_replica.cc:331] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.415637 1186 raft_consensus.cc:2241] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.415691 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 68f2ccd8a7a64b89b44f2d6767ec3466 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.417912 1186 raft_consensus.cc:2270] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.418601 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_68b569f22b1245008a2bd45d0be4d8cf" } modify_external_catalogs: true
I20250905 19:52:18.418803 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_68b569f22b1245008a2bd45d0be4d8cf" } modify_external_catalogs: true
I20250905 19:52:18.421420 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_56e29cf512764408b243635ae01a6ab0" } modify_external_catalogs: true
I20250905 19:52:18.422166 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_56e29cf512764408b243635ae01a6ab0" } modify_external_catalogs: true
I20250905 19:52:18.423002 1169 ts_tablet_manager.cc:1428] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250905 19:52:18.423509 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_332f3b495a6548caa4cc12a01588ee6d" } modify_external_catalogs: true
I20250905 19:52:18.423676 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_332f3b495a6548caa4cc12a01588ee6d" } modify_external_catalogs: true
I20250905 19:52:18.424583 1169 tablet_bootstrap.cc:492] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.427342 1169 tablet_bootstrap.cc:654] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.428730 1186 ts_tablet_manager.cc:1905] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.431412 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0de80dfdff2546a2a39368585f13a394" } modify_external_catalogs: true
I20250905 19:52:18.431645 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0de80dfdff2546a2a39368585f13a394" } modify_external_catalogs: true
I20250905 19:52:18.435987 1413 raft_consensus.cc:491] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.436160 1413 raft_consensus.cc:513] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.436867 1413 leader_election.cc:290] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.437579 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf2881d228964cd3bef040a06045d8d2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.438190 848 leader_election.cc:343] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.416512 1170 tablet_bootstrap.cc:492] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.439040 1170 ts_tablet_manager.cc:1397] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.078s user 0.008s sys 0.001s
I20250905 19:52:18.440517 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf2881d228964cd3bef040a06045d8d2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.440747 1052 raft_consensus.cc:2466] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:18.441130 849 leader_election.cc:304] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.441569 1413 raft_consensus.cc:2802] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:18.441699 1413 raft_consensus.cc:491] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:18.441797 1413 raft_consensus.cc:3058] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.444021 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:18.444597 448 catalog_manager.cc:6652] ReplaceTablet: tablet 302053e1611b4e22a9a2e7b60ae68f27 deleted and replaced by tablet ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:18.445271 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.445350 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:18.445266 1413 raft_consensus.cc:513] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.445859 447 catalog_manager.cc:6652] ReplaceTablet: tablet 57f851a6eecc4a41806bdaa19cc5afab deleted and replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
I20250905 19:52:18.445927 1413 leader_election.cc:290] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.446077 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 57259d8eb0d44d1cbac80047c7ad67b6 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.446306 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.446394 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab with delete_type TABLET_DATA_DELETED (replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.446467 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf2881d228964cd3bef040a06045d8d2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.446669 1052 raft_consensus.cc:3058] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:18.447149 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 302053e1611b4e22a9a2e7b60ae68f27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302053e1611b4e22a9a2e7b60ae68f27 already in progress: creating tablet
W20250905 19:52:18.447388 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 57f851a6eecc4a41806bdaa19cc5afab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f851a6eecc4a41806bdaa19cc5afab already in progress: creating tablet
I20250905 19:52:18.447736 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab with delete_type TABLET_DATA_DELETED (replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.448561 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 9e421213b7bd4dd8a03515519395de7d
I20250905 19:52:18.449008 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:18.449409 1052 raft_consensus.cc:2466] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:18.449455 455 catalog_manager.cc:6652] ReplaceTablet: tablet 68f2ccd8a7a64b89b44f2d6767ec3466 deleted and replaced by tablet ab4f69c45295422f85b310aa55220e9b
I20250905 19:52:18.449769 849 leader_election.cc:304] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:18.450237 1413 raft_consensus.cc:2802] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.450464 1394 raft_consensus.cc:491] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.450620 1394 raft_consensus.cc:513] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.450721 1413 raft_consensus.cc:695] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:18.451042 1413 consensus_queue.cc:237] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.451292 1394 leader_election.cc:290] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.451860 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.452805 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19e4edddafcd4d22a0e0ae66b0b953e0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:18.452929 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f2ccd8a7a64b89b44f2d6767ec3466 already in progress: creating tablet
W20250905 19:52:18.453279 986 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.453430 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e421213b7bd4dd8a03515519395de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.453692 1185 tablet_replica.cc:331] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:18.453898 1185 raft_consensus.cc:2241] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.453883 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.454119 1185 raft_consensus.cc:2270] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.454562 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e421213b7bd4dd8a03515519395de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.455338 1185 ts_tablet_manager.cc:1905] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.457185 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e421213b7bd4dd8a03515519395de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.457628 1184 ts_tablet_manager.cc:1918] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.457576 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab with delete_type TABLET_DATA_DELETED (replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.457660 1464 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:18.457840 1184 log.cc:1199] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a3ae86cb11e24d348e2f12e982f3ab9f
I20250905 19:52:18.458338 1184 ts_tablet_manager.cc:1939] T a3ae86cb11e24d348e2f12e982f3ab9f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.459489 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cf2881d228964cd3bef040a06045d8d2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:18.459332 1169 tablet_bootstrap.cc:492] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.460327 1169 ts_tablet_manager.cc:1397] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.003s
I20250905 19:52:18.460733 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4786a32c4c014dac926b49908380da96"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.461200 848 leader_election.cc:343] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.461195 1169 raft_consensus.cc:357] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.461359 1169 raft_consensus.cc:383] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.461447 1169 raft_consensus.cc:738] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.461736 1169 consensus_queue.cc:260] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.461848 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_cddec42a8c1c415ba27927b52d055a08" } modify_external_catalogs: true
I20250905 19:52:18.461956 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7240f6ab8dbf4c16a2651dfc5a92d15b" } new_table_name: "default.Table_68f98d1c0de04ff9a6ad5188ab275ec9" modify_external_catalogs: true
I20250905 19:52:18.462025 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_cddec42a8c1c415ba27927b52d055a08" } modify_external_catalogs: true
I20250905 19:52:18.463790 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19e4edddafcd4d22a0e0ae66b0b953e0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.464588 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4786a32c4c014dac926b49908380da96 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.464592 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:18.464761 984 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.464849 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.464882 984 leader_election.cc:304] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.465883 1394 raft_consensus.cc:2747] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.467072 1170 raft_consensus.cc:357] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.467355 1170 raft_consensus.cc:383] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.467483 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_aa894f644dbe444bbe04d396b9ceb056"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:18.468469 1169 ts_tablet_manager.cc:1428] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
W20250905 19:52:18.468554 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aa894f644dbe444bbe04d396b9ceb056 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.471619 1170 raft_consensus.cc:738] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.472002 1170 consensus_queue.cc:260] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.473258 1169 tablet_bootstrap.cc:492] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.475476 1170 ts_tablet_manager.cc:1428] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.036s user 0.001s sys 0.002s
I20250905 19:52:18.476293 1169 tablet_bootstrap.cc:654] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.476730 1170 tablet_bootstrap.cc:492] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.479128 1185 ts_tablet_manager.cc:1918] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:18.481648 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a3ae86cb11e24d348e2f12e982f3ab9f (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]) successfully deleted
I20250905 19:52:18.481976 1184 tablet_replica.cc:331] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.482149 1170 tablet_bootstrap.cc:654] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.482211 1184 raft_consensus.cc:2241] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.482376 1184 raft_consensus.cc:2270] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.482996 1184 ts_tablet_manager.cc:1905] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.482987 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 9e9994863c374354838aa0f817d0057e
I20250905 19:52:18.484318 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.486284 453 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]). Expected version 1 got 0
I20250905 19:52:18.487718 1262 fs_manager.cc:730] Time spent opening block manager: real 0.085s user 0.004s sys 0.000s
I20250905 19:52:18.487900 1262 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
uuid: "e38333af29db4bf6af41ee5a25d6701e"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:18.484562 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.490525 1185 log.cc:1199] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9e421213b7bd4dd8a03515519395de7d
I20250905 19:52:18.491237 1185 ts_tablet_manager.cc:1939] T 9e421213b7bd4dd8a03515519395de7d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:18.492583 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9e421213b7bd4dd8a03515519395de7d (table default.Table_68b569f22b1245008a2bd45d0be4d8cf [id=152edbecfe3648659c48a2cf89ff9798]) successfully deleted
I20250905 19:52:18.494307 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8e07fab3c1d94464a864b8aa58a9372d" } new_table_name: "default.Table_f43b3348e3084812974970a576a6fd15" modify_external_catalogs: true
W20250905 19:52:18.496217 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 9e9994863c374354838aa0f817d0057e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9994863c374354838aa0f817d0057e already in progress: creating tablet
I20250905 19:52:18.500968 1169 tablet_bootstrap.cc:492] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.500998 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:18.501154 1169 ts_tablet_manager.cc:1397] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.000s
I20250905 19:52:18.501574 450 catalog_manager.cc:6652] ReplaceTablet: tablet 3602094b96c14e66b997e8035fc53c40 deleted and replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:18.502256 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:18.502494 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:18.502903 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 8ed6109bdf8f4caf8c223a42401b94ab consensus state has no leader
I20250905 19:52:18.503095 1171 tablet_bootstrap.cc:492] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.503257 1171 ts_tablet_manager.cc:1397] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.141s user 0.003s sys 0.033s
I20250905 19:52:18.503768 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.504076 1171 raft_consensus.cc:357] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.504292 1171 raft_consensus.cc:383] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.504273 1169 raft_consensus.cc:357] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.504367 1171 raft_consensus.cc:738] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.504460 1169 raft_consensus.cc:383] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.504525 1169 raft_consensus.cc:738] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.504666 1171 consensus_queue.cc:260] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.504786 1169 consensus_queue.cc:260] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.504916 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.505187 888 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.505424 1171 ts_tablet_manager.cc:1428] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:18.505734 1171 tablet_bootstrap.cc:492] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:18.505714 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 3602094b96c14e66b997e8035fc53c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3602094b96c14e66b997e8035fc53c40 already in progress: creating tablet
W20250905 19:52:18.506047 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet f28e403dc2f6471ebea1805fd843cfb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28e403dc2f6471ebea1805fd843cfb1 already in progress: creating tablet
I20250905 19:52:18.507647 1169 ts_tablet_manager.cc:1428] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:18.488945 1262 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:18.509052 1171 tablet_bootstrap.cc:654] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.512025 1169 tablet_bootstrap.cc:492] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.512380 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_5163f5e51d334e59a090e55589bc90dd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.513038 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5163f5e51d334e59a090e55589bc90dd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.514678 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:18.516036 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.517527 1169 tablet_bootstrap.cc:654] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.518239 1394 raft_consensus.cc:491] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.518396 1394 raft_consensus.cc:513] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.519021 1394 leader_election.cc:290] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.519848 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59c13a5b598a4597931ddf5d476aec85" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:18.520252 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:18.520382 986 leader_election.cc:343] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.520833 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 60a41dbd8782460cb0d3c5f8bd59354f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60a41dbd8782460cb0d3c5f8bd59354f already in progress: creating tablet
I20250905 19:52:18.526868 1184 ts_tablet_manager.cc:1918] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.527071 1184 log.cc:1199] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:18.527575 1184 ts_tablet_manager.cc:1939] T 6d12c7fbb84849c68274810b3fdcba04 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:18.527956 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 29c2b030fafd44ad9e648b2527a4ad07 consensus state has no leader
I20250905 19:52:18.530504 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59c13a5b598a4597931ddf5d476aec85" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.531229 984 leader_election.cc:343] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.531370 984 leader_election.cc:304] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.531810 1394 raft_consensus.cc:2747] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.532364 453 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]). Expected version 1 got 0
I20250905 19:52:18.532644 753 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.533123 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.533507 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.533839 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.535706 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6d12c7fbb84849c68274810b3fdcba04 (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
W20250905 19:52:18.536422 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 53ac46ce05694bc2b53e195638be9e23 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 53ac46ce05694bc2b53e195638be9e23 already in progress: creating tablet
W20250905 19:52:18.537400 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 4fe94d1254d04949aa25346d8e323c3b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4fe94d1254d04949aa25346d8e323c3b already in progress: creating tablet
W20250905 19:52:18.538651 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4d2ed1bc09c94526bc9662f2d4d21f8f already in progress: creating tablet
W20250905 19:52:18.539382 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 00d8144bb4f14358af539b7aae326918 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 00d8144bb4f14358af539b7aae326918 already in progress: creating tablet
I20250905 19:52:18.540447 1186 ts_tablet_manager.cc:1918] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.540599 1186 log.cc:1199] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d23d6df63c1a4e02a43bccf625e200bb
W20250905 19:52:18.540609 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 302053e1611b4e22a9a2e7b60ae68f27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302053e1611b4e22a9a2e7b60ae68f27 already in progress: creating tablet
I20250905 19:52:18.540952 1184 tablet_replica.cc:331] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.541023 1186 ts_tablet_manager.cc:1939] T d23d6df63c1a4e02a43bccf625e200bb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.541165 1184 raft_consensus.cc:2241] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.541353 1184 raft_consensus.cc:2270] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.541893 1184 ts_tablet_manager.cc:1905] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.542184 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d23d6df63c1a4e02a43bccf625e200bb (table default.Table_a77e895ae1804560879140ca64db74f6 [id=2573b827e5224e5cabab22ba1d9aa3cb]) successfully deleted
I20250905 19:52:18.542589 1186 tablet_replica.cc:331] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.542830 1186 raft_consensus.cc:2241] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.543087 1186 raft_consensus.cc:2270] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.544399 1186 ts_tablet_manager.cc:1905] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.552451 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (table default.Table_8c4a8cf3d2354147a7c24ecab6f27c05 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]). Expected version 1 got 0
I20250905 19:52:18.553172 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_d6e9e58d90324cd2a7110d05c7588160 [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 2 got 0
I20250905 19:52:18.557073 1171 tablet_bootstrap.cc:492] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.557368 1171 ts_tablet_manager.cc:1397] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.052s user 0.005s sys 0.003s
I20250905 19:52:18.558384 1171 raft_consensus.cc:357] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.558561 1171 raft_consensus.cc:383] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.558652 1171 raft_consensus.cc:738] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.558753 888 tablet_service.cc:1468] Processing CreateTablet for tablet 091ef981d6b54756830927ea111e78aa (DEFAULT_TABLE table=default.Table_79a0ddd169324b64b69f90b3d65bfa11 [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.558904 1171 consensus_queue.cc:260] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.559268 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091ef981d6b54756830927ea111e78aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.562484 1171 ts_tablet_manager.cc:1428] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:18.564327 886 tablet_service.cc:1468] Processing CreateTablet for tablet ba161282480e4736889559a2e37b195f (DEFAULT_TABLE table=default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.566424 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 091ef981d6b54756830927ea111e78aa (DEFAULT_TABLE table=default.Table_79a0ddd169324b64b69f90b3d65bfa11 [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.566893 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091ef981d6b54756830927ea111e78aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.567627 1027 tablet_service.cc:1468] Processing CreateTablet for tablet a3805127dd78489e8545da7414ead904 (DEFAULT_TABLE table=default.Table_e65ca281d8804277a83ab7bfc2ab9cca [id=3ed6b530762e4290aa5c45ee1a55c100]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.568154 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3805127dd78489e8545da7414ead904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.569280 1186 ts_tablet_manager.cc:1918] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.569416 1186 log.cc:1199] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d7d7567807594201a598d715ffbb318b
I20250905 19:52:18.569449 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ba161282480e4736889559a2e37b195f (DEFAULT_TABLE table=default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.569837 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba161282480e4736889559a2e37b195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.570888 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 54628965b82e470d811eff95ffc56c5f (DEFAULT_TABLE table=default.Table_e304b44cbe8f4c77bae2af66f9b5920e [id=0b36fe8f4d004271aa9bbe78a44dc583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.571194 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54628965b82e470d811eff95ffc56c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.571269 1024 tablet_service.cc:1468] Processing CreateTablet for tablet de19c897d3314fecb17cedb3370eed93 (DEFAULT_TABLE table=default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.571712 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de19c897d3314fecb17cedb3370eed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.575022 750 tablet_service.cc:1468] Processing CreateTablet for tablet 091ef981d6b54756830927ea111e78aa (DEFAULT_TABLE table=default.Table_79a0ddd169324b64b69f90b3d65bfa11 [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.575559 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 091ef981d6b54756830927ea111e78aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.576367 744 tablet_service.cc:1468] Processing CreateTablet for tablet ba161282480e4736889559a2e37b195f (DEFAULT_TABLE table=default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.576671 746 tablet_service.cc:1468] Processing CreateTablet for tablet a3805127dd78489e8545da7414ead904 (DEFAULT_TABLE table=default.Table_e65ca281d8804277a83ab7bfc2ab9cca [id=3ed6b530762e4290aa5c45ee1a55c100]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.576802 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba161282480e4736889559a2e37b195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.577188 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3805127dd78489e8545da7414ead904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.580173 753 tablet_service.cc:1468] Processing CreateTablet for tablet de19c897d3314fecb17cedb3370eed93 (DEFAULT_TABLE table=default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.580703 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de19c897d3314fecb17cedb3370eed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.582316 743 tablet_service.cc:1468] Processing CreateTablet for tablet 54628965b82e470d811eff95ffc56c5f (DEFAULT_TABLE table=default.Table_e304b44cbe8f4c77bae2af66f9b5920e [id=0b36fe8f4d004271aa9bbe78a44dc583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.582737 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54628965b82e470d811eff95ffc56c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.576371 1186 ts_tablet_manager.cc:1939] T d7d7567807594201a598d715ffbb318b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.584851 1186 tablet_replica.cc:331] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.585047 1186 raft_consensus.cc:2241] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.585199 1186 raft_consensus.cc:2270] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.586012 1186 ts_tablet_manager.cc:1905] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.588579 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d7d7567807594201a598d715ffbb318b (table default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
I20250905 19:52:18.594488 1186 ts_tablet_manager.cc:1918] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.594667 1186 log.cc:1199] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/305a7758527d4d52a669b2f2c4ceedbb
I20250905 19:52:18.595098 1186 ts_tablet_manager.cc:1939] T 305a7758527d4d52a669b2f2c4ceedbb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.596084 1169 tablet_bootstrap.cc:492] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.596241 1169 ts_tablet_manager.cc:1397] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.088s user 0.005s sys 0.005s
I20250905 19:52:18.597076 1169 raft_consensus.cc:357] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.597270 1169 raft_consensus.cc:383] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.597280 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 305a7758527d4d52a669b2f2c4ceedbb (table default.Table_5996e867324248b1b6c68868f6dfe8e2 [id=2bf888448df6476f818a8efe70480c4b]) successfully deleted
I20250905 19:52:18.597359 1169 raft_consensus.cc:738] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:18.597611 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9e421213b7bd4dd8a03515519395de7d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e421213b7bd4dd8a03515519395de7d already in progress: creating tablet
I20250905 19:52:18.597632 1169 consensus_queue.cc:260] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:18.597846 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 57f851a6eecc4a41806bdaa19cc5afab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 57f851a6eecc4a41806bdaa19cc5afab already in progress: creating tablet
W20250905 19:52:18.598101 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f2ccd8a7a64b89b44f2d6767ec3466 already in progress: creating tablet
W20250905 19:52:18.598336 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 302053e1611b4e22a9a2e7b60ae68f27 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 302053e1611b4e22a9a2e7b60ae68f27 already in progress: creating tablet
W20250905 19:52:18.598573 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet f28e403dc2f6471ebea1805fd843cfb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28e403dc2f6471ebea1805fd843cfb1 already in progress: creating tablet
I20250905 19:52:18.599339 1169 ts_tablet_manager.cc:1428] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:18.599647 1169 tablet_bootstrap.cc:492] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.603091 1184 ts_tablet_manager.cc:1918] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.603219 1184 log.cc:1199] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:18.603613 1184 ts_tablet_manager.cc:1939] T 57f851a6eecc4a41806bdaa19cc5afab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.604640 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 57f851a6eecc4a41806bdaa19cc5afab (table default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
I20250905 19:52:18.605226 1169 tablet_bootstrap.cc:654] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:18.606637 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 68f2ccd8a7a64b89b44f2d6767ec3466 already in progress: creating tablet
I20250905 19:52:18.606953 1184 tablet_replica.cc:331] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.607149 1184 raft_consensus.cc:2241] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:18.607463 1184 raft_consensus.cc:2270] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.609412 1184 ts_tablet_manager.cc:1905] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:18.613267 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9e9994863c374354838aa0f817d0057e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9994863c374354838aa0f817d0057e already in progress: creating tablet
I20250905 19:52:18.613727 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f542b700d5364bfeb6e1e0111834fab0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.614537 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f542b700d5364bfeb6e1e0111834fab0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.616252 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4ec0491dbe8841049cceaeee7c721211"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.616969 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ec0491dbe8841049cceaeee7c721211 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.617111 1169 tablet_bootstrap.cc:492] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.617280 1169 ts_tablet_manager.cc:1397] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250905 19:52:18.618141 1169 raft_consensus.cc:357] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.618346 1169 raft_consensus.cc:383] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.618414 1169 raft_consensus.cc:738] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.618481 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8c4a8cf3d2354147a7c24ecab6f27c05" } new_table_name: "default.Table_9a54ba37e2a941ecaf67d8e56bc940f9" modify_external_catalogs: true
I20250905 19:52:18.618711 1169 consensus_queue.cc:260] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.619575 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_580d1de7c8164c1c8c1e3ba556bd5c94" } new_table_name: "default.Table_ca7a00374cab4386a886afd1d3a90342" modify_external_catalogs: true
I20250905 19:52:18.621510 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.622028 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.622184 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_93a9ab7d352a4a3da35219aa6f20f432"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.622658 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_93a9ab7d352a4a3da35219aa6f20f432 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.623132 1184 ts_tablet_manager.cc:1918] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:18.623227 1169 ts_tablet_manager.cc:1428] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:18.623282 1184 log.cc:1199] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9e421213b7bd4dd8a03515519395de7d
I20250905 19:52:18.623708 1184 ts_tablet_manager.cc:1939] T 9e421213b7bd4dd8a03515519395de7d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:18.623737 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_fb47213914d74250866aa0cb93a06647"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.623901 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 60a41dbd8782460cb0d3c5f8bd59354f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60a41dbd8782460cb0d3c5f8bd59354f already in progress: creating tablet
W20250905 19:52:18.624171 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3602094b96c14e66b997e8035fc53c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3602094b96c14e66b997e8035fc53c40 already in progress: creating tablet
W20250905 19:52:18.624269 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fb47213914d74250866aa0cb93a06647 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.628432 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ba161282480e4736889559a2e37b195f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.628984 1171 tablet_bootstrap.cc:492] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.631912 878 tablet_service.cc:1468] Processing CreateTablet for tablet de19c897d3314fecb17cedb3370eed93 (DEFAULT_TABLE table=default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.632416 1169 tablet_bootstrap.cc:492] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.632717 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet de19c897d3314fecb17cedb3370eed93. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.633005 1170 tablet_bootstrap.cc:492] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.633183 1170 ts_tablet_manager.cc:1397] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.157s user 0.002s sys 0.002s
I20250905 19:52:18.633473 885 tablet_service.cc:1468] Processing CreateTablet for tablet 54628965b82e470d811eff95ffc56c5f (DEFAULT_TABLE table=default.Table_e304b44cbe8f4c77bae2af66f9b5920e [id=0b36fe8f4d004271aa9bbe78a44dc583]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.634057 889 tablet_service.cc:1468] Processing CreateTablet for tablet a3805127dd78489e8545da7414ead904 (DEFAULT_TABLE table=default.Table_e65ca281d8804277a83ab7bfc2ab9cca [id=3ed6b530762e4290aa5c45ee1a55c100]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.634413 1170 raft_consensus.cc:357] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.634632 1170 raft_consensus.cc:383] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.634716 1170 raft_consensus.cc:738] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.634953 1170 consensus_queue.cc:260] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.635114 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 54628965b82e470d811eff95ffc56c5f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.635370 1169 tablet_bootstrap.cc:654] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.635479 1170 ts_tablet_manager.cc:1428] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:18.635792 1170 tablet_bootstrap.cc:492] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.637560 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3805127dd78489e8545da7414ead904. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.638252 1170 tablet_bootstrap.cc:654] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.640244 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9e421213b7bd4dd8a03515519395de7d (table default.Table_68b569f22b1245008a2bd45d0be4d8cf [id=152edbecfe3648659c48a2cf89ff9798]) successfully deleted
W20250905 19:52:18.648860 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 9e9994863c374354838aa0f817d0057e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9e9994863c374354838aa0f817d0057e already in progress: creating tablet
W20250905 19:52:18.649801 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 3602094b96c14e66b997e8035fc53c40 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3602094b96c14e66b997e8035fc53c40 already in progress: creating tablet
W20250905 19:52:18.650086 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet f28e403dc2f6471ebea1805fd843cfb1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f28e403dc2f6471ebea1805fd843cfb1 already in progress: creating tablet
W20250905 19:52:18.650296 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 60a41dbd8782460cb0d3c5f8bd59354f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 60a41dbd8782460cb0d3c5f8bd59354f already in progress: creating tablet
I20250905 19:52:18.657608 1413 raft_consensus.cc:491] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.657886 1413 raft_consensus.cc:513] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.658718 1413 leader_election.cc:290] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.659633 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f8d3cbdda0c94e139259d90919dc2f95" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:18.659950 1169 tablet_bootstrap.cc:492] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.660125 1169 ts_tablet_manager.cc:1397] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.007s
W20250905 19:52:18.660420 848 leader_election.cc:343] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.661134 1169 raft_consensus.cc:357] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.661311 1169 raft_consensus.cc:383] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.661415 1169 raft_consensus.cc:738] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:18.661723 849 leader_election.cc:343] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: f8d3cbdda0c94e139259d90919dc2f95
I20250905 19:52:18.661679 1169 consensus_queue.cc:260] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.661846 849 leader_election.cc:304] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.662189 1169 ts_tablet_manager.cc:1428] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:18.662473 1169 tablet_bootstrap.cc:492] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.662524 1413 raft_consensus.cc:2747] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.664294 1169 tablet_bootstrap.cc:654] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.671435 1169 tablet_bootstrap.cc:492] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.671639 1169 ts_tablet_manager.cc:1397] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.000s
I20250905 19:52:18.672508 1169 raft_consensus.cc:357] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.672734 1169 raft_consensus.cc:383] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.672820 1169 raft_consensus.cc:738] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.673063 1169 consensus_queue.cc:260] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.673445 1171 tablet_bootstrap.cc:654] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.675113 1169 ts_tablet_manager.cc:1428] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:18.675426 1169 tablet_bootstrap.cc:492] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.677246 1169 tablet_bootstrap.cc:654] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.686254 1394 raft_consensus.cc:491] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.686473 1394 raft_consensus.cc:513] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.687193 1394 leader_election.cc:290] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.687271 1169 tablet_bootstrap.cc:492] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.687397 1169 ts_tablet_manager.cc:1397] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20250905 19:52:18.687953 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77cbdda0e2d641bd8092f00925bb9a9c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:18.688508 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "77cbdda0e2d641bd8092f00925bb9a9c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.688585 986 leader_election.cc:343] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.689066 984 leader_election.cc:343] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.689198 984 leader_election.cc:304] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.689221 1169 raft_consensus.cc:357] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.689370 1169 raft_consensus.cc:383] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.689457 1169 raft_consensus.cc:738] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.689687 1169 consensus_queue.cc:260] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.689996 1236 raft_consensus.cc:2747] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.690225 1169 ts_tablet_manager.cc:1428] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:18.692082 1169 tablet_bootstrap.cc:492] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.693749 1169 tablet_bootstrap.cc:654] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.702445 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9fe38a14d69846b8b468fdcef3fff0f5" } new_table_name: "default.Table_37d80cf1db464cff9b77c0aff7a595c2" modify_external_catalogs: true
I20250905 19:52:18.704856 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 09b46a12076e492e89d0c0178bcf0524 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.705405 1186 tablet_replica.cc:331] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.705675 1186 raft_consensus.cc:2241] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.705986 1186 raft_consensus.cc:2270] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.706887 1186 ts_tablet_manager.cc:1905] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.715368 1186 ts_tablet_manager.cc:1918] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.715516 1186 log.cc:1199] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/09b46a12076e492e89d0c0178bcf0524
I20250905 19:52:18.715924 1186 ts_tablet_manager.cc:1939] T 09b46a12076e492e89d0c0178bcf0524 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:18.718514 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0b1ff79095aa492b843efd3cc222a0db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.719282 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0b1ff79095aa492b843efd3cc222a0db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.720378 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 09b46a12076e492e89d0c0178bcf0524 (table default.Table_2cf8f5ed2b6e4f6fb2208f0df2ac193a [id=3b8967eb30ad46cdbf272136f23318f1]) successfully deleted
W20250905 19:52:18.732844 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4fac0e1f483b40ab8d66cb5a9ee808fc consensus state has no leader
I20250905 19:52:18.740859 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8e27522da15a4d8ebb55104bcbe5fc5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.741752 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8e27522da15a4d8ebb55104bcbe5fc5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:18.750411 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03a12cf719f34096bb82f5846bbe3b2d has no consensus state
I20250905 19:52:18.752740 1171 tablet_bootstrap.cc:492] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.752979 1171 ts_tablet_manager.cc:1397] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.187s user 0.003s sys 0.006s
I20250905 19:52:18.754637 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_48447ad862474825a10962298c97650d" } modify_external_catalogs: true
I20250905 19:52:18.754678 1171 raft_consensus.cc:357] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.754914 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_48447ad862474825a10962298c97650d" } modify_external_catalogs: true
I20250905 19:52:18.755005 1171 raft_consensus.cc:383] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.755089 1171 raft_consensus.cc:738] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.755447 1171 consensus_queue.cc:260] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.759385 1171 ts_tablet_manager.cc:1428] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.004s sys 0.000s
I20250905 19:52:18.760596 1171 tablet_bootstrap.cc:492] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.763036 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 6d12c7fbb84849c68274810b3fdcba04 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.763592 1171 tablet_bootstrap.cc:654] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.766526 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_191f4f9b06184a8a8429cce5802c520e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.767273 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_191f4f9b06184a8a8429cce5802c520e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.769505 1394 raft_consensus.cc:491] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.769702 1394 raft_consensus.cc:513] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.770365 1394 leader_election.cc:290] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.770841 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8a32f742eaf473e9f6f9bf214f53679" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:18.770967 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8a32f742eaf473e9f6f9bf214f53679" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.771520 986 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.771776 984 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.771901 984 leader_election.cc:304] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.772387 1394 raft_consensus.cc:2747] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.776737 1394 raft_consensus.cc:491] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.776921 1394 raft_consensus.cc:513] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.777613 1394 leader_election.cc:290] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.778772 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d480531ba94f4b0f9fe40e88fe91512e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:18.778767 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_68f98d1c0de04ff9a6ad5188ab275ec9" } modify_external_catalogs: true
I20250905 19:52:18.778947 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_68f98d1c0de04ff9a6ad5188ab275ec9" } modify_external_catalogs: true
I20250905 19:52:18.779258 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d480531ba94f4b0f9fe40e88fe91512e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:18.779918 986 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.780436 984 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.780565 984 leader_election.cc:304] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.781042 1394 raft_consensus.cc:2747] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.781891 1169 tablet_bootstrap.cc:492] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.782050 1169 ts_tablet_manager.cc:1397] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.090s user 0.010s sys 0.013s
I20250905 19:52:18.782806 1169 raft_consensus.cc:357] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.783020 1169 raft_consensus.cc:383] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.783103 1169 raft_consensus.cc:738] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.783931 1169 consensus_queue.cc:260] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.784469 1169 ts_tablet_manager.cc:1428] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:18.787420 1169 tablet_bootstrap.cc:492] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.788622 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5ce48afe3ba94518bd44699d763ac7a9" } modify_external_catalogs: true
I20250905 19:52:18.788870 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5ce48afe3ba94518bd44699d763ac7a9" } modify_external_catalogs: true
I20250905 19:52:18.789932 1169 tablet_bootstrap.cc:654] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.792080 1170 tablet_bootstrap.cc:492] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.792248 1170 ts_tablet_manager.cc:1397] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.157s user 0.026s sys 0.052s
I20250905 19:52:18.793107 1170 raft_consensus.cc:357] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.793296 1170 raft_consensus.cc:383] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.793403 1170 raft_consensus.cc:738] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.793678 1170 consensus_queue.cc:260] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.795810 1170 ts_tablet_manager.cc:1428] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:18.796175 1394 raft_consensus.cc:491] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.796340 1394 raft_consensus.cc:513] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.797050 1394 leader_election.cc:290] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.797449 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7edcf57f7753449d8a5fcdb6f4f1dc21" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:18.797714 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7edcf57f7753449d8a5fcdb6f4f1dc21" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:18.798259 986 leader_election.cc:343] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.798666 1170 tablet_bootstrap.cc:492] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:18.798802 984 leader_election.cc:343] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.798916 984 leader_election.cc:304] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.799379 1394 raft_consensus.cc:2747] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.799885 886 tablet_service.cc:1468] Processing CreateTablet for tablet a8b6823580cb4040b49ed0a49b748965 (DEFAULT_TABLE table=default.Table_aa894f644dbe444bbe04d396b9ceb056 [id=4172afda1ef0438cb4f4695bb8340e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.800431 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b6823580cb4040b49ed0a49b748965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.800536 1170 tablet_bootstrap.cc:654] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.801014 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]). Expected version 1 got 0
I20250905 19:52:18.801396 454 catalog_manager.cc:5582] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:18.801962 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:18.802580 888 tablet_service.cc:1468] Processing CreateTablet for tablet 6e15350150db413fa4f100171280f46e (DEFAULT_TABLE table=default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.802947 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e15350150db413fa4f100171280f46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.804922 1394 raft_consensus.cc:491] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.805071 1394 raft_consensus.cc:513] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.805658 1394 leader_election.cc:290] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:18.805809 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 1c6295c8c7874df89662b8cef6459fec
I20250905 19:52:18.806382 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fac0e1f483b40ab8d66cb5a9ee808fc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:18.806718 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540] alter complete (version 1)
W20250905 19:52:18.807008 984 leader_election.cc:343] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.809080 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6e15350150db413fa4f100171280f46e (DEFAULT_TABLE table=default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.809564 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e15350150db413fa4f100171280f46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.809612 1030 tablet_service.cc:1468] Processing CreateTablet for tablet a8b6823580cb4040b49ed0a49b748965 (DEFAULT_TABLE table=default.Table_aa894f644dbe444bbe04d396b9ceb056 [id=4172afda1ef0438cb4f4695bb8340e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.809695 1029 tablet_service.cc:1468] Processing CreateTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e (DEFAULT_TABLE table=default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.809937 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b6823580cb4040b49ed0a49b748965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.810143 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.810441 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad (DEFAULT_TABLE table=default.Table_5163f5e51d334e59a090e55589bc90dd [id=0a06a88771a94e60b5bb57fa7f7d8215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.810796 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1a8e7059b348cba34069ef49bf1aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.815199 1032 tablet_service.cc:1468] Processing CreateTablet for tablet ac5c88cf8d15433a940e937f0a225582 (DEFAULT_TABLE table=default.Table_5918e71853e2403a9d36c683ed58ac88 [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.815580 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c88cf8d15433a940e937f0a225582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.816205 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fac0e1f483b40ab8d66cb5a9ee808fc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:18.816876 986 leader_election.cc:343] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.817056 986 leader_election.cc:304] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.817377 1394 raft_consensus.cc:2747] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.819610 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d (DEFAULT_TABLE table=default.Table_ca7a00374cab4386a886afd1d3a90342 [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.821257 1170 tablet_bootstrap.cc:492] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.821421 1170 ts_tablet_manager.cc:1397] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.023s user 0.006s sys 0.000s
I20250905 19:52:18.822227 1170 raft_consensus.cc:357] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.822444 1170 raft_consensus.cc:383] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.822551 1170 raft_consensus.cc:738] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.823035 743 tablet_service.cc:1468] Processing CreateTablet for tablet ac5c88cf8d15433a940e937f0a225582 (DEFAULT_TABLE table=default.Table_5918e71853e2403a9d36c683ed58ac88 [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.823140 744 tablet_service.cc:1468] Processing CreateTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d (DEFAULT_TABLE table=default.Table_ca7a00374cab4386a886afd1d3a90342 [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.823356 750 tablet_service.cc:1468] Processing CreateTablet for tablet 6e15350150db413fa4f100171280f46e (DEFAULT_TABLE table=default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.823498 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c88cf8d15433a940e937f0a225582. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.823648 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e15350150db413fa4f100171280f46e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.823709 739 tablet_service.cc:1468] Processing CreateTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e (DEFAULT_TABLE table=default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.824164 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c6295c8c7874df89662b8cef6459fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.824258 753 tablet_service.cc:1468] Processing CreateTablet for tablet a8b6823580cb4040b49ed0a49b748965 (DEFAULT_TABLE table=default.Table_aa894f644dbe444bbe04d396b9ceb056 [id=4172afda1ef0438cb4f4695bb8340e66]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.824580 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.824724 746 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad (DEFAULT_TABLE table=default.Table_5163f5e51d334e59a090e55589bc90dd [id=0a06a88771a94e60b5bb57fa7f7d8215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.826525 879 tablet_service.cc:1468] Processing CreateTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e (DEFAULT_TABLE table=default.Table_e88b4d40afe0429daf95a8650aee6126 [id=9049c258ecc54305be2e9641520fb9ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.826983 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.829398 885 tablet_service.cc:1468] Processing CreateTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d (DEFAULT_TABLE table=default.Table_ca7a00374cab4386a886afd1d3a90342 [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.829946 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a12cf719f34096bb82f5846bbe3b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.832336 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a12cf719f34096bb82f5846bbe3b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.832240 1170 consensus_queue.cc:260] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.833462 1170 ts_tablet_manager.cc:1428] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250905 19:52:18.833797 1170 tablet_bootstrap.cc:492] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.839085 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c6295c8c7874df89662b8cef6459fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.839407 1185 tablet_replica.cc:331] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:18.840173 1170 tablet_bootstrap.cc:654] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.840410 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_0be185ca49de49f09cc109dc19ec7063" } new_table_name: "default.Table_885d880838c44eab9ff1220768e59373" modify_external_catalogs: true
I20250905 19:52:18.842314 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8b6823580cb4040b49ed0a49b748965. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.842927 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1a8e7059b348cba34069ef49bf1aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.847656 1170 tablet_bootstrap.cc:492] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.847784 743 tablet_service.cc:1468] Processing CreateTablet for tablet ab4f69c45295422f85b310aa55220e9b (DEFAULT_TABLE table=default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.847904 1413 raft_consensus.cc:491] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.848285 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4f69c45295422f85b310aa55220e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.848531 1413 raft_consensus.cc:513] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.849248 1413 leader_election.cc:290] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.847925 878 tablet_service.cc:1468] Processing CreateTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad (DEFAULT_TABLE table=default.Table_5163f5e51d334e59a090e55589bc90dd [id=0a06a88771a94e60b5bb57fa7f7d8215]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.850533 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c884954e5dc4445a12252953593d0bc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:18.851172 1185 raft_consensus.cc:2241] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:18.851416 1185 pending_rounds.cc:70] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:18.851691 1185 pending_rounds.cc:77] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089538822926336 op_type: NO_OP noop_request { }
I20250905 19:52:18.851825 1185 raft_consensus.cc:2887] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:18.852567 1185 raft_consensus.cc:2270] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:18.852802 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 1c6295c8c7874df89662b8cef6459fec because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1c6295c8c7874df89662b8cef6459fec already in progress: creating tablet
I20250905 19:52:18.852963 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1f1a8e7059b348cba34069ef49bf1aad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.853106 889 tablet_service.cc:1468] Processing CreateTablet for tablet ac5c88cf8d15433a940e937f0a225582 (DEFAULT_TABLE table=default.Table_5918e71853e2403a9d36c683ed58ac88 [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.853504 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac5c88cf8d15433a940e937f0a225582. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:18.859892 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05ec559d0f83445892f8285f9bdeb2c8 consensus state has no leader
I20250905 19:52:18.860091 880 tablet_service.cc:1468] Processing CreateTablet for tablet ab4f69c45295422f85b310aa55220e9b (DEFAULT_TABLE table=default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:18.864045 848 leader_election.cc:343] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.864616 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4f69c45295422f85b310aa55220e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.868227 1413 raft_consensus.cc:491] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.868376 1413 raft_consensus.cc:513] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.869063 1413 leader_election.cc:290] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.870105 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:18.870669 848 leader_election.cc:343] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.871371 1185 ts_tablet_manager.cc:1905] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.877979 1185 ts_tablet_manager.cc:1918] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:18.878403 1203 raft_consensus.cc:491] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:18.878618 1203 raft_consensus.cc:513] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.878865 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c5d8d96a9bb34b5ea9b36388f41faf80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:18.879375 1203 leader_election.cc:290] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:18.879432 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c5d8d96a9bb34b5ea9b36388f41faf80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.880990 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f681ca53501741449436b37d7ee502c7" } new_table_name: "default.Table_4447fb37960e42dfa2c6eb8ced9c6b68" modify_external_catalogs: true
I20250905 19:52:18.880599 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:18.887267 1185 log.cc:1199] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/1c6295c8c7874df89662b8cef6459fec
I20250905 19:52:18.887773 1185 ts_tablet_manager.cc:1939] T 1c6295c8c7874df89662b8cef6459fec P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:18.889058 1171 tablet_bootstrap.cc:492] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:18.889211 1171 ts_tablet_manager.cc:1397] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.129s user 0.018s sys 0.026s
I20250905 19:52:18.889555 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 1c6295c8c7874df89662b8cef6459fec (table default.Table_0de80dfdff2546a2a39368585f13a394 [id=2151f6f3348c4d1fb376448ecfcbae8a]) successfully deleted
I20250905 19:52:18.890081 1171 raft_consensus.cc:357] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.890259 1171 raft_consensus.cc:383] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.890345 1171 raft_consensus.cc:738] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.847826 1170 ts_tablet_manager.cc:1397] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.004s
I20250905 19:52:18.890559 1171 consensus_queue.cc:260] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.892086 1170 raft_consensus.cc:357] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.892283 1170 raft_consensus.cc:383] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.892403 1170 raft_consensus.cc:738] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.893627 1170 consensus_queue.cc:260] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.894428 1171 ts_tablet_manager.cc:1428] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:18.895732 1170 ts_tablet_manager.cc:1428] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:18.896754 1170 tablet_bootstrap.cc:492] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.898475 1171 tablet_bootstrap.cc:492] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:18.900202 1171 tablet_bootstrap.cc:654] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.900810 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03a12cf719f34096bb82f5846bbe3b2d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.904631 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_be613708cf304dc2856d98a4e1f6920f" } modify_external_catalogs: true
I20250905 19:52:18.904882 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_be613708cf304dc2856d98a4e1f6920f" } modify_external_catalogs: true
I20250905 19:52:18.907225 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:18.908035 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 43111a3512fb4fc0bdf4ee245b16695b from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.910077 452 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 2 got 0
I20250905 19:52:18.910699 1170 tablet_bootstrap.cc:654] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.911186 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7fa399abb0ac4fbeaeb856e415850864"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.911891 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7fa399abb0ac4fbeaeb856e415850864 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.918246 1170 tablet_bootstrap.cc:492] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:18.918432 1170 ts_tablet_manager.cc:1397] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.001s
I20250905 19:52:18.919256 1170 raft_consensus.cc:357] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.920670 1170 raft_consensus.cc:383] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.920778 1170 raft_consensus.cc:738] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.921008 1170 consensus_queue.cc:260] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.921501 1170 ts_tablet_manager.cc:1428] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:18.921808 1170 tablet_bootstrap.cc:492] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:18.924813 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c6295c8c7874df89662b8cef6459fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.924044 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c884954e5dc4445a12252953593d0bc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.926604 1170 tablet_bootstrap.cc:654] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.927112 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "09be30768f5742cf9a4bf1e3fe86dfe8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.928005 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:18.928203 1053 raft_consensus.cc:2466] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:18.928228 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:18.928253 711 leader_election.cc:304] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:18.928686 1203 raft_consensus.cc:2747] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:18.929368 451 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041] alter complete (version 1)
I20250905 19:52:18.930233 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 091ef981d6b54756830927ea111e78aa from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.931118 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 42f3c1efb94e446fb0128b6b412821e2 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.932204 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c6295c8c7874df89662b8cef6459fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.932545 849 leader_election.cc:304] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.932996 1413 raft_consensus.cc:2802] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:18.933104 1413 raft_consensus.cc:491] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:18.933223 1413 raft_consensus.cc:3058] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.934623 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.934841 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab with delete_type TABLET_DATA_DELETED (replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.934960 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.935053 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.936053 1028 tablet_service.cc:1468] Processing CreateTablet for tablet ab4f69c45295422f85b310aa55220e9b (DEFAULT_TABLE table=default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:18.936292 1052 raft_consensus.cc:2466] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:18.936792 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 1c6295c8c7874df89662b8cef6459fec because the tablet was not found. No further retry: Not found: Tablet not found: 1c6295c8c7874df89662b8cef6459fec
I20250905 19:52:18.937088 849 leader_election.cc:304] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:18.937367 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 44769f1170354870ae2d34fdb67ffbae consensus state has no leader
I20250905 19:52:18.937486 1419 raft_consensus.cc:2802] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:18.937588 1419 raft_consensus.cc:491] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:18.937665 1419 raft_consensus.cc:3058] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.938508 1184 tablet_replica.cc:331] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:18.938709 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:18.939738 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4c76077e4adf45a7adc736b0e69032eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c76077e4adf45a7adc736b0e69032eb already in progress: creating tablet
I20250905 19:52:18.940513 1419 raft_consensus.cc:513] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.940814 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a884eca0e54a44dc920114e61a199689
I20250905 19:52:18.941431 448 master_service.cc:948] ReplaceTablet: received request to replace tablet 57259d8eb0d44d1cbac80047c7ad67b6 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:18.941514 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c884954e5dc4445a12252953593d0bc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:18.941644 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a884eca0e54a44dc920114e61a199689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.941680 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.941923 1185 tablet_replica.cc:331] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:18.942157 1185 raft_consensus.cc:2241] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.941514 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a884eca0e54a44dc920114e61a199689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.942341 1185 raft_consensus.cc:2270] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:18.942179 848 leader_election.cc:343] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:18.942881 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a884eca0e54a44dc920114e61a199689 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a884eca0e54a44dc920114e61a199689 already in progress: creating tablet
I20250905 19:52:18.942997 1413 raft_consensus.cc:513] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.943768 1413 leader_election.cc:290] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:18.944343 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ab4f69c45295422f85b310aa55220e9b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:18.944623 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:18.944675 1419 leader_election.cc:290] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:18.945137 848 leader_election.cc:343] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:18.945494 1185 ts_tablet_manager.cc:1905] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.945659 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:18.946419 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8ed6109bdf8f4caf8c223a42401b94ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed6109bdf8f4caf8c223a42401b94ab already in progress: creating tablet
I20250905 19:52:18.947419 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.947986 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c20eea1b024d7f93f18f0c089efc42) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:18.953148 1185 ts_tablet_manager.cc:1918] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.953296 1185 log.cc:1199] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a884eca0e54a44dc920114e61a199689
I20250905 19:52:18.953294 1184 raft_consensus.cc:2241] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.953658 1185 ts_tablet_manager.cc:1939] T a884eca0e54a44dc920114e61a199689 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:18.954725 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a884eca0e54a44dc920114e61a199689 (table default.Table_5ce48afe3ba94518bd44699d763ac7a9 [id=8c67898a816046e084be67f08f5edb71]) successfully deleted
W20250905 19:52:18.955454 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 4c76077e4adf45a7adc736b0e69032eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c76077e4adf45a7adc736b0e69032eb already in progress: creating tablet
I20250905 19:52:18.956069 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c5807c1132747e6af79d93e7496d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:18.956516 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8ed6109bdf8f4caf8c223a42401b94ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed6109bdf8f4caf8c223a42401b94ab already in progress: creating tablet
I20250905 19:52:18.956979 1185 tablet_replica.cc:331] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:18.958108 1185 raft_consensus.cc:2241] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.958420 1185 raft_consensus.cc:2270] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.959028 1185 ts_tablet_manager.cc:1905] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.960294 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a884eca0e54a44dc920114e61a199689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:18.961592 447 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 2 got 0
I20250905 19:52:18.961798 447 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:18.963405 1184 raft_consensus.cc:2270] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.964215 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7d2ebfe52fb74920893aea327256e248"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:18.964929 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7d2ebfe52fb74920893aea327256e248 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:18.966859 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2c884954e5dc4445a12252953593d0bc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.967049 1052 raft_consensus.cc:3058] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.967701 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:18.969527 1184 ts_tablet_manager.cc:1905] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.970212 449 catalog_manager.cc:5582] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:18.974231 1052 raft_consensus.cc:2466] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:18.967872 1053 raft_consensus.cc:3058] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:18.974647 849 leader_election.cc:304] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.975111 1419 raft_consensus.cc:2802] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.975244 1419 raft_consensus.cc:695] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:18.975519 1419 consensus_queue.cc:237] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.977138 1053 raft_consensus.cc:2466] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:18.977468 849 leader_election.cc:304] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:18.978005 1413 raft_consensus.cc:2802] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:18.983325 1184 ts_tablet_manager.cc:1918] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:18.979902 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:18.986727 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e421213b7bd4dd8a03515519395de7d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:18.987735 1186 tablet_replica.cc:331] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:18.988022 1186 raft_consensus.cc:2241] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:18.988232 1186 raft_consensus.cc:2270] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:18.989271 1186 ts_tablet_manager.cc:1905] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:18.988621 1169 tablet_bootstrap.cc:492] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:18.991225 1169 ts_tablet_manager.cc:1397] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.204s user 0.003s sys 0.051s
I20250905 19:52:18.992000 1169 raft_consensus.cc:357] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.992172 1169 raft_consensus.cc:383] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:18.992252 1169 raft_consensus.cc:738] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:18.992475 1169 consensus_queue.cc:260] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:18.994844 1169 ts_tablet_manager.cc:1428] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:18.995141 1169 tablet_bootstrap.cc:492] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:18.996870 1169 tablet_bootstrap.cc:654] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:18.997676 1184 log.cc:1199] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/1c6295c8c7874df89662b8cef6459fec
I20250905 19:52:18.998056 1184 ts_tablet_manager.cc:1939] T 1c6295c8c7874df89662b8cef6459fec P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.001497 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 1c6295c8c7874df89662b8cef6459fec (table default.Table_0de80dfdff2546a2a39368585f13a394 [id=2151f6f3348c4d1fb376448ecfcbae8a]) successfully deleted
W20250905 19:52:19.002133 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 8ed6109bdf8f4caf8c223a42401b94ab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8ed6109bdf8f4caf8c223a42401b94ab already in progress: creating tablet
W20250905 19:52:19.002604 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 57f851a6eecc4a41806bdaa19cc5afab because the tablet was not found. No further retry: Not found: Tablet not found: 57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:19.002866 1184 tablet_replica.cc:331] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.003062 1184 raft_consensus.cc:2241] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.003201 1184 raft_consensus.cc:2270] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.003695 1184 ts_tablet_manager.cc:1905] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.003856 1262 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:19.005034 1262 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:19.005339 1262 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:19.009116 1170 tablet_bootstrap.cc:492] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.009285 1170 ts_tablet_manager.cc:1397] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.088s user 0.000s sys 0.009s
I20250905 19:52:19.010532 1169 tablet_bootstrap.cc:492] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.010722 1169 ts_tablet_manager.cc:1397] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
I20250905 19:52:19.011133 1171 tablet_bootstrap.cc:492] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.011313 1171 ts_tablet_manager.cc:1397] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.113s user 0.005s sys 0.010s
I20250905 19:52:19.011545 1169 raft_consensus.cc:357] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.011736 1169 raft_consensus.cc:383] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.011804 1169 raft_consensus.cc:738] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.012090 1169 consensus_queue.cc:260] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.012203 1171 raft_consensus.cc:357] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.012395 1171 raft_consensus.cc:383] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.012480 1171 raft_consensus.cc:738] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.013491 1171 consensus_queue.cc:260] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.015049 1169 ts_tablet_manager.cc:1428] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:19.015352 1169 tablet_bootstrap.cc:492] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.018492 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ab4f69c45295422f85b310aa55220e9b
I20250905 19:52:19.019364 739 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.020709 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.021252 1170 raft_consensus.cc:357] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.021431 1170 raft_consensus.cc:383] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.021523 1170 raft_consensus.cc:738] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.022142 1170 consensus_queue.cc:260] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.022696 1186 ts_tablet_manager.cc:1918] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.022922 1186 log.cc:1199] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9e421213b7bd4dd8a03515519395de7d
I20250905 19:52:19.023407 1186 ts_tablet_manager.cc:1939] T 9e421213b7bd4dd8a03515519395de7d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:19.024466 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9e421213b7bd4dd8a03515519395de7d (table default.Table_68b569f22b1245008a2bd45d0be4d8cf [id=152edbecfe3648659c48a2cf89ff9798]) successfully deleted
W20250905 19:52:19.025041 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ab4f69c45295422f85b310aa55220e9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab4f69c45295422f85b310aa55220e9b already in progress: creating tablet
I20250905 19:52:19.025223 1169 tablet_bootstrap.cc:654] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.026671 1170 ts_tablet_manager.cc:1428] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250905 19:52:19.028493 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5918e71853e2403a9d36c683ed58ac88" } new_table_name: "default.Table_44c46e2e89984ccfabdeee5a59adbf7b" modify_external_catalogs: true
I20250905 19:52:19.029321 1170 tablet_bootstrap.cc:492] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:19.031224 1170 tablet_bootstrap.cc:654] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.031301 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5163f5e51d334e59a090e55589bc90dd" } new_table_name: "default.Table_7186efc2caef4940ade686830c963ba1" modify_external_catalogs: true
I20250905 19:52:19.032842 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a414c05c2c0c40b8a55a7776f36ecdb7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.033455 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.034396 1184 ts_tablet_manager.cc:1918] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.034569 1184 log.cc:1199] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/7edcf57f7753449d8a5fcdb6f4f1dc21
I20250905 19:52:19.034971 1184 ts_tablet_manager.cc:1939] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.036199 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 (table default.Table_b86b0c6f5fe4496ead402827d2db3f1d [id=8638b095c62741559b2281befee964ea]) successfully deleted
I20250905 19:52:19.036548 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 1c6295c8c7874df89662b8cef6459fec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.041877 1186 tablet_replica.cc:331] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:19.042062 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:19.042639 455 catalog_manager.cc:6652] ReplaceTablet: tablet 43111a3512fb4fc0bdf4ee245b16695b deleted and replaced by tablet efb70327f55844879f89449a7116010f
I20250905 19:52:19.042840 1186 raft_consensus.cc:2241] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.043025 1186 raft_consensus.cc:2270] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.044734 1184 tablet_replica.cc:331] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.044987 1184 raft_consensus.cc:2241] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.045192 1184 raft_consensus.cc:2270] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.045874 1184 ts_tablet_manager.cc:1905] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.046030 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:19.047955 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 4c76077e4adf45a7adc736b0e69032eb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 4c76077e4adf45a7adc736b0e69032eb already in progress: creating tablet
I20250905 19:52:19.048436 888 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.048681 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.051626 1169 tablet_bootstrap.cc:492] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.051811 1169 ts_tablet_manager.cc:1397] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20250905 19:52:19.052734 1394 raft_consensus.cc:491] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.052879 1394 raft_consensus.cc:513] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.053571 1394 leader_election.cc:290] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.054531 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "087172093fff4c41980dc2daeabb07c3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.054795 1171 ts_tablet_manager.cc:1428] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.043s user 0.008s sys 0.000s
W20250905 19:52:19.055687 984 leader_election.cc:343] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.056959 1169 raft_consensus.cc:357] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.057235 1169 raft_consensus.cc:383] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.057376 1169 raft_consensus.cc:738] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.057711 1169 consensus_queue.cc:260] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.060405 1169 ts_tablet_manager.cc:1428] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:19.060633 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1c6571236cd7434d98aec149d06dc6b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.060794 1169 tablet_bootstrap.cc:492] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:19.061306 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c6571236cd7434d98aec149d06dc6b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.063023 1413 raft_consensus.cc:695] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:19.063100 1169 tablet_bootstrap.cc:654] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.070369 1170 tablet_bootstrap.cc:492] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.070711 1170 ts_tablet_manager.cc:1397] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.001s
I20250905 19:52:19.072120 1170 raft_consensus.cc:357] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.072489 1170 raft_consensus.cc:383] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.072651 1170 raft_consensus.cc:738] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.073231 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_10627325344c469294d0ea3e9036daea"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.073374 1170 consensus_queue.cc:260] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:19.073958 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10627325344c469294d0ea3e9036daea in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.078413 1394 raft_consensus.cc:491] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.078645 1394 raft_consensus.cc:513] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.079407 1394 leader_election.cc:290] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.080426 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9477afb5b47c46d4b24128a6c4bdabc6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.081061 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9477afb5b47c46d4b24128a6c4bdabc6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.081698 984 leader_election.cc:343] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:19.082163 986 leader_election.cc:343] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.082280 986 leader_election.cc:304] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.082765 1394 raft_consensus.cc:2747] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.083638 455 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]). Expected version 1 got 0
I20250905 19:52:19.084432 1185 ts_tablet_manager.cc:1918] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.084585 1185 log.cc:1199] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:19.084993 1185 ts_tablet_manager.cc:1939] T 57259d8eb0d44d1cbac80047c7ad67b6 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.086277 1495 raft_consensus.cc:491] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.086428 1495 raft_consensus.cc:513] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.087456 1495 leader_election.cc:290] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.086464 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 57259d8eb0d44d1cbac80047c7ad67b6 (table default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]) successfully deleted
W20250905 19:52:19.088050 849 leader_election.cc:343] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 59faa6716fed4ea6b563df5f024b2058
I20250905 19:52:19.088150 1170 ts_tablet_manager.cc:1428] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
W20250905 19:52:19.088286 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ab4f69c45295422f85b310aa55220e9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab4f69c45295422f85b310aa55220e9b already in progress: creating tablet
I20250905 19:52:19.089985 1170 tablet_bootstrap.cc:492] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:19.090673 1184 ts_tablet_manager.cc:1918] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.090823 1184 log.cc:1199] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/7c5807c1132747e6af79d93e7496d500
I20250905 19:52:19.090981 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59faa6716fed4ea6b563df5f024b2058" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.091297 1184 ts_tablet_manager.cc:1939] T 7c5807c1132747e6af79d93e7496d500 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:19.092032 848 leader_election.cc:343] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.092173 848 leader_election.cc:304] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.092592 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 7c5807c1132747e6af79d93e7496d500 (table default.Table_f3bcb1e148584db3b4336af7b8028437 [id=cd492ea5b4b4434b960ba37fcf479450]) successfully deleted
I20250905 19:52:19.094851 1184 tablet_replica.cc:331] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.095135 1184 raft_consensus.cc:2241] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.095335 1184 raft_consensus.cc:2270] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.095378 1170 tablet_bootstrap.cc:654] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.096037 1184 ts_tablet_manager.cc:1905] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.096297 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.096628 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "087172093fff4c41980dc2daeabb07c3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:19.097095 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 43111a3512fb4fc0bdf4ee245b16695b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43111a3512fb4fc0bdf4ee245b16695b already in progress: creating tablet
I20250905 19:52:19.097205 1413 consensus_queue.cc:237] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.098198 1171 tablet_bootstrap.cc:492] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:19.097180 986 leader_election.cc:343] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.100688 986 leader_election.cc:304] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.101167 1394 raft_consensus.cc:2747] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.101329 1419 raft_consensus.cc:2747] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.102041 1171 tablet_bootstrap.cc:654] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.103120 775 raft_consensus.cc:3058] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.104453 1186 ts_tablet_manager.cc:1905] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.108466 1504 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:19.109565 1505 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:19.110812 1186 ts_tablet_manager.cc:1918] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.110941 1186 log.cc:1199] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/1c6295c8c7874df89662b8cef6459fec
I20250905 19:52:19.110921 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9067a9bea2cc48ecab23b9fd1749474c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.111361 1186 ts_tablet_manager.cc:1939] T 1c6295c8c7874df89662b8cef6459fec P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:19.111622 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9067a9bea2cc48ecab23b9fd1749474c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.112498 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 1c6295c8c7874df89662b8cef6459fec (table default.Table_0de80dfdff2546a2a39368585f13a394 [id=2151f6f3348c4d1fb376448ecfcbae8a]) successfully deleted
W20250905 19:52:19.114099 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ac5c88cf8d15433a940e937f0a225582 consensus state has no leader
W20250905 19:52:19.115104 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 43111a3512fb4fc0bdf4ee245b16695b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 43111a3512fb4fc0bdf4ee245b16695b already in progress: creating tablet
W20250905 19:52:19.115183 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1f1a8e7059b348cba34069ef49bf1aad consensus state has no leader
I20250905 19:52:19.115463 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 2683a36998674f739b41d2a57339954c from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:19.115876 1203 raft_consensus.cc:491] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.116075 1203 raft_consensus.cc:513] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.116796 1203 leader_election.cc:290] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.117093 1171 tablet_bootstrap.cc:492] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.117254 1171 ts_tablet_manager.cc:1397] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.062s user 0.001s sys 0.002s
I20250905 19:52:19.117522 453 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]). Expected version 1 got 0
I20250905 19:52:19.118131 1171 raft_consensus.cc:357] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.118340 1171 raft_consensus.cc:383] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.118425 1171 raft_consensus.cc:738] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.118508 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff" } modify_external_catalogs: true
I20250905 19:52:19.118690 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbab0209a48d4ad8b2f2dec0562e9598" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.118700 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff" } modify_external_catalogs: true
I20250905 19:52:19.119240 1169 tablet_bootstrap.cc:492] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.119405 1170 tablet_bootstrap.cc:492] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.119346 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbab0209a48d4ad8b2f2dec0562e9598" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:19.119422 1169 ts_tablet_manager.cc:1397] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.059s user 0.005s sys 0.005s
I20250905 19:52:19.119554 1170 ts_tablet_manager.cc:1397] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.006s
I20250905 19:52:19.120016 711 leader_election.cc:304] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:19.120278 1169 raft_consensus.cc:357] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.120436 1203 raft_consensus.cc:2747] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.120469 1169 raft_consensus.cc:383] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.120399 1170 raft_consensus.cc:357] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.120535 1169 raft_consensus.cc:738] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.120589 1170 raft_consensus.cc:383] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.120677 1170 raft_consensus.cc:738] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.120800 1169 consensus_queue.cc:260] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.121284 1169 ts_tablet_manager.cc:1428] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:19.122855 1169 tablet_bootstrap.cc:492] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.124256 1170 consensus_queue.cc:260] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.125622 1169 tablet_bootstrap.cc:654] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.125962 1170 ts_tablet_manager.cc:1428] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:19.127406 1184 ts_tablet_manager.cc:1918] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.128825 1170 tablet_bootstrap.cc:492] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:19.129047 1184 log.cc:1199] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a884eca0e54a44dc920114e61a199689
I20250905 19:52:19.133744 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_14d9939868624cd49dc6e2facdc4eb08" } new_table_name: "default.Table_4074e68bdbd84b8ab0734c59a600b1b7" modify_external_catalogs: true
I20250905 19:52:19.134965 1184 ts_tablet_manager.cc:1939] T a884eca0e54a44dc920114e61a199689 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.136518 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a884eca0e54a44dc920114e61a199689 (table default.Table_5ce48afe3ba94518bd44699d763ac7a9 [id=8c67898a816046e084be67f08f5edb71]) successfully deleted
I20250905 19:52:19.137833 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a8a69461d6db47e298856ec5bc896165"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.138545 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a8a69461d6db47e298856ec5bc896165 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.143476 1170 tablet_bootstrap.cc:654] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.144019 1171 consensus_queue.cc:260] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.144660 1171 ts_tablet_manager.cc:1428] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.027s user 0.000s sys 0.003s
I20250905 19:52:19.144816 1169 tablet_bootstrap.cc:492] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.145776 1169 ts_tablet_manager.cc:1397] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.007s
I20250905 19:52:19.146134 1508 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250905 19:52:19.146265 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ab4f69c45295422f85b310aa55220e9b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ab4f69c45295422f85b310aa55220e9b already in progress: creating tablet
I20250905 19:52:19.148015 1169 raft_consensus.cc:357] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.148198 1169 raft_consensus.cc:383] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.148275 1169 raft_consensus.cc:738] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.148558 1184 tablet_replica.cc:331] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.148806 1184 raft_consensus.cc:2241] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.149022 1184 raft_consensus.cc:2270] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.149618 1184 ts_tablet_manager.cc:1905] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.151226 1171 tablet_bootstrap.cc:492] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.151721 1169 consensus_queue.cc:260] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.152976 1171 tablet_bootstrap.cc:654] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.153051 1413 consensus_queue.cc:1035] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:19.154454 1394 raft_consensus.cc:491] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.154610 1394 raft_consensus.cc:513] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.155241 1394 leader_election.cc:290] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.156064 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6f837fb7293474b8ad357f52e42e42e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.156749 984 leader_election.cc:343] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.157426 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6f837fb7293474b8ad357f52e42e42e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:19.157943 986 leader_election.cc:343] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.158370 986 leader_election.cc:304] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.159052 1394 raft_consensus.cc:2747] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.159188 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e9cdd0c9bd8840658a9c9aeb2385265a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.159834 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9cdd0c9bd8840658a9c9aeb2385265a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.156724 1169 ts_tablet_manager.cc:1428] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250905 19:52:19.164328 1169 tablet_bootstrap.cc:492] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.166410 1508 tablet.cc:1722] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:19.172080 774 raft_consensus.cc:3058] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.172094 776 raft_consensus.cc:3058] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.176947 447 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]). Expected version 2 got 0
I20250905 19:52:19.182787 1171 tablet_bootstrap.cc:492] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.182988 1171 ts_tablet_manager.cc:1397] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.004s
I20250905 19:52:19.183090 449 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4] alter complete (version 1)
I20250905 19:52:19.184078 1169 tablet_bootstrap.cc:654] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.189059 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d (DEFAULT_TABLE table=default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff [id=8f425647eea7481181ba823eb955d1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.189569 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e5ca7bd3e84916a11e01afbf48171d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.189365 888 tablet_service.cc:1468] Processing CreateTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b (DEFAULT_TABLE table=default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.190001 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578a6b9224cc4d8585d6c4b241762e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.192409 886 tablet_service.cc:1468] Processing CreateTablet for tablet 8edf4b54974c427a855eee5792d75292 (DEFAULT_TABLE table=default.Table_93a9ab7d352a4a3da35219aa6f20f432 [id=b5acc86aee9545d99f8d8332c3c76a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.192890 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8edf4b54974c427a855eee5792d75292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.193217 1184 ts_tablet_manager.cc:1918] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.193346 1184 log.cc:1199] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:19.193728 1184 ts_tablet_manager.cc:1939] T 43111a3512fb4fc0bdf4ee245b16695b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.196416 885 tablet_service.cc:1468] Processing CreateTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 (DEFAULT_TABLE table=default.Table_4ec0491dbe8841049cceaeee7c721211 [id=23b4c9dd0ca7459fa0d0a8b5f7033fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.196928 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06850faa25de4b3d83f6bf98ccd370f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.200404 889 tablet_service.cc:1468] Processing CreateTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 (DEFAULT_TABLE table=default.Table_f542b700d5364bfeb6e1e0111834fab0 [id=27f7071a498d467aa9047b07dac76b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.200879 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46ea498d6a14d1ebdffea415cfc7114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.183815 1171 raft_consensus.cc:357] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.201218 1171 raft_consensus.cc:383] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.201340 1171 raft_consensus.cc:738] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.202692 1171 consensus_queue.cc:260] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.204416 879 tablet_service.cc:1468] Processing CreateTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d (DEFAULT_TABLE table=default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff [id=8f425647eea7481181ba823eb955d1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.204836 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e5ca7bd3e84916a11e01afbf48171d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.208492 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b (DEFAULT_TABLE table=default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.209095 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578a6b9224cc4d8585d6c4b241762e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.211119 1171 ts_tablet_manager.cc:1428] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20250905 19:52:19.212457 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 8edf4b54974c427a855eee5792d75292 (DEFAULT_TABLE table=default.Table_93a9ab7d352a4a3da35219aa6f20f432 [id=b5acc86aee9545d99f8d8332c3c76a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.213032 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8edf4b54974c427a855eee5792d75292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.213423 1171 tablet_bootstrap.cc:492] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.216357 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 (DEFAULT_TABLE table=default.Table_4ec0491dbe8841049cceaeee7c721211 [id=23b4c9dd0ca7459fa0d0a8b5f7033fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.216910 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06850faa25de4b3d83f6bf98ccd370f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.220266 1495 consensus_queue.cc:1035] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:19.220402 1024 tablet_service.cc:1468] Processing CreateTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 (DEFAULT_TABLE table=default.Table_f542b700d5364bfeb6e1e0111834fab0 [id=27f7071a498d467aa9047b07dac76b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.220830 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46ea498d6a14d1ebdffea415cfc7114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.234166 1171 tablet_bootstrap.cc:654] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:19.236472 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_37d80cf1db464cff9b77c0aff7a595c2 [id=f5891226b675435f81cbdca3c8b4f43e], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 05ec559d0f83445892f8285f9bdeb2c8 consensus state has no leader
I20250905 19:52:19.237497 878 tablet_service.cc:1515] Processing DeleteTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f with delete_type TABLET_DATA_DELETED (replaced by tablet 544dffc8b77049c295921d0af091b17d) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.237794 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 1664cbe248514535953a1ef2d312f98f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.238025 1185 tablet_replica.cc:331] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.238149 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 59faa6716fed4ea6b563df5f024b2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 05ec559d0f83445892f8285f9bdeb2c8) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.238263 1185 raft_consensus.cc:2241] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.238341 878 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.238445 1185 raft_consensus.cc:2270] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.238554 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 43111a3512fb4fc0bdf4ee245b16695b (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]) successfully deleted
I20250905 19:52:19.237991 893 tablet_service.cc:1515] Processing DeleteTablet for tablet a884eca0e54a44dc920114e61a199689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.239604 1185 ts_tablet_manager.cc:1905] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.242810 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.243054 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.243256 1184 tablet_replica.cc:331] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.243470 1184 raft_consensus.cc:2241] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.243664 1184 raft_consensus.cc:2270] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.244124 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.244155 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.244283 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.244349 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.244887 1184 ts_tablet_manager.cc:1905] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.245977 1185 ts_tablet_manager.cc:1918] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.246131 1185 log.cc:1199] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a3ae86cb11e24d348e2f12e982f3ab9f
I20250905 19:52:19.246227 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet e6f837fb7293474b8ad357f52e42e42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.246501 1185 ts_tablet_manager.cc:1939] T a3ae86cb11e24d348e2f12e982f3ab9f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.246506 1508 tablet.cc:1722] T 85c648f799304fa9a94bec665a2dcf97 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:19.247534 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a3ae86cb11e24d348e2f12e982f3ab9f (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]) successfully deleted
I20250905 19:52:19.248735 1185 tablet_replica.cc:331] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.248992 1185 raft_consensus.cc:2241] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.251109 1184 ts_tablet_manager.cc:1918] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.251240 1184 log.cc:1199] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/0edac849e8834894be2d4ff323edd13d
I20250905 19:52:19.251579 1184 ts_tablet_manager.cc:1939] T 0edac849e8834894be2d4ff323edd13d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.252652 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 0edac849e8834894be2d4ff323edd13d (table default.Table_a39be9af778c4fa49c2376cb4bd7397b [id=e9ed839c21da4467a0f165e7fedbe966]) successfully deleted
I20250905 19:52:19.252995 1184 tablet_replica.cc:331] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.253185 1184 raft_consensus.cc:2241] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.253381 1184 raft_consensus.cc:2270] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.253990 1184 ts_tablet_manager.cc:1905] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.255100 1171 tablet_bootstrap.cc:492] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.255254 1171 ts_tablet_manager.cc:1397] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.042s user 0.002s sys 0.002s
I20250905 19:52:19.257257 1169 tablet_bootstrap.cc:492] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.257433 1169 ts_tablet_manager.cc:1397] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.093s user 0.010s sys 0.037s
I20250905 19:52:19.258221 1169 raft_consensus.cc:357] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.258425 1169 raft_consensus.cc:383] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.258508 1169 raft_consensus.cc:738] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.258798 1169 consensus_queue.cc:260] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.264583 1169 ts_tablet_manager.cc:1428] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250905 19:52:19.268313 1169 tablet_bootstrap.cc:492] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.269737 1203 raft_consensus.cc:491] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.269914 1203 raft_consensus.cc:513] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.270546 1203 leader_election.cc:290] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.271855 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:19.272145 1052 raft_consensus.cc:2391] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:19.272892 1169 tablet_bootstrap.cc:654] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.275074 1184 ts_tablet_manager.cc:1918] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.275277 1184 log.cc:1199] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:19.275713 1184 ts_tablet_manager.cc:1939] T a8a32f742eaf473e9f6f9bf214f53679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.276805 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a8a32f742eaf473e9f6f9bf214f53679 (table default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]) successfully deleted
I20250905 19:52:19.277205 1184 tablet_replica.cc:331] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.277452 1184 raft_consensus.cc:2241] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.277791 1184 raft_consensus.cc:2270] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.278532 1184 ts_tablet_manager.cc:1905] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.287070 1169 tablet_bootstrap.cc:492] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.287235 1169 ts_tablet_manager.cc:1397] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.004s
I20250905 19:52:19.287882 1169 raft_consensus.cc:357] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.288285 1169 raft_consensus.cc:383] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.288398 1169 raft_consensus.cc:738] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.288690 1169 consensus_queue.cc:260] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.291074 1184 ts_tablet_manager.cc:1918] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.291280 1184 log.cc:1199] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:19.291862 1184 ts_tablet_manager.cc:1939] T 757a7d3fba8d43dda36474539b7296ba P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.292383 1185 raft_consensus.cc:2270] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.292819 1171 raft_consensus.cc:357] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.292984 1185 ts_tablet_manager.cc:1905] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.293020 1171 raft_consensus.cc:383] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.293123 1171 raft_consensus.cc:738] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.293362 1171 consensus_queue.cc:260] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.293932 1171 ts_tablet_manager.cc:1428] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250905 19:52:19.294235 1171 tablet_bootstrap.cc:492] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.296933 1171 tablet_bootstrap.cc:654] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.298250 1495 raft_consensus.cc:491] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.298436 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "813f208881ac4782874138987a710d93" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.298611 1491 raft_consensus.cc:491] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.298764 1491 raft_consensus.cc:513] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.299417 1491 leader_election.cc:290] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.300216 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c8a8848304491bbd4a7971f96322f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.300899 848 leader_election.cc:343] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.298403 1495 raft_consensus.cc:513] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.301587 1495 leader_election.cc:290] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.302456 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "76a5c85dbda140d882a5a63a13aaabd7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.302982 848 leader_election.cc:343] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.303730 774 raft_consensus.cc:3058] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.304260 712 leader_election.cc:304] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:19.305629 1491 consensus_queue.cc:1035] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:19.306080 1491 consensus_queue.cc:1035] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:19.306175 1171 tablet_bootstrap.cc:492] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.306375 1171 ts_tablet_manager.cc:1397] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.003s
I20250905 19:52:19.306951 1203 raft_consensus.cc:2747] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.307209 1171 raft_consensus.cc:357] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.307612 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c8a8848304491bbd4a7971f96322f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:19.307718 1171 raft_consensus.cc:383] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.307793 1171 raft_consensus.cc:738] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.307829 1052 raft_consensus.cc:2466] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:19.307991 1495 consensus_queue.cc:1035] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:19.308143 1170 tablet_bootstrap.cc:492] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.308064 1171 consensus_queue.cc:260] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.308295 1170 ts_tablet_manager.cc:1397] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.180s user 0.004s sys 0.000s
I20250905 19:52:19.308849 1171 ts_tablet_manager.cc:1428] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:19.309283 849 leader_election.cc:304] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.309335 1169 ts_tablet_manager.cc:1428] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.021s user 0.003s sys 0.000s
I20250905 19:52:19.309434 1171 tablet_bootstrap.cc:492] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.309665 1169 tablet_bootstrap.cc:492] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.309643 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 757a7d3fba8d43dda36474539b7296ba (table default.Table_a5a47a603b674a41b47e7bf4b55b6a1b [id=106f635aee6b4cd4a7f7b1af9088a286]) successfully deleted
I20250905 19:52:19.309919 1491 raft_consensus.cc:2802] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:19.310031 1491 raft_consensus.cc:491] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:19.310130 1491 raft_consensus.cc:3058] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:19.311023 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:41317, user_credentials={real_user=slave}} blocked reactor thread for 142387us
I20250905 19:52:19.311602 1171 tablet_bootstrap.cc:654] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.312839 739 tablet_service.cc:1468] Processing CreateTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b (DEFAULT_TABLE table=default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.313323 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 578a6b9224cc4d8585d6c4b241762e1b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.313261 744 tablet_service.cc:1468] Processing CreateTablet for tablet 8edf4b54974c427a855eee5792d75292 (DEFAULT_TABLE table=default.Table_93a9ab7d352a4a3da35219aa6f20f432 [id=b5acc86aee9545d99f8d8332c3c76a8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.313357 743 tablet_service.cc:1468] Processing CreateTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d (DEFAULT_TABLE table=default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff [id=8f425647eea7481181ba823eb955d1e1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.313663 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8edf4b54974c427a855eee5792d75292. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.313877 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e5ca7bd3e84916a11e01afbf48171d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.314144 746 tablet_service.cc:1468] Processing CreateTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 (DEFAULT_TABLE table=default.Table_4ec0491dbe8841049cceaeee7c721211 [id=23b4c9dd0ca7459fa0d0a8b5f7033fd9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.314522 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 06850faa25de4b3d83f6bf98ccd370f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.316283 753 tablet_service.cc:1468] Processing CreateTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 (DEFAULT_TABLE table=default.Table_f542b700d5364bfeb6e1e0111834fab0 [id=27f7071a498d467aa9047b07dac76b64]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.316694 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c46ea498d6a14d1ebdffea415cfc7114. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.316742 1185 ts_tablet_manager.cc:1918] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.316852 1185 log.cc:1199] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/1664cbe248514535953a1ef2d312f98f
I20250905 19:52:19.317188 1185 ts_tablet_manager.cc:1939] T 1664cbe248514535953a1ef2d312f98f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.318118 1185 tablet_replica.cc:331] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.318322 1185 raft_consensus.cc:2241] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.318466 1185 raft_consensus.cc:2270] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.318987 1185 ts_tablet_manager.cc:1905] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.320638 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 1664cbe248514535953a1ef2d312f98f (table default.Table_6a14b2be50f942d8bfd8c47247512879 [id=353aa71a6ee64c41aa94daadd629d845]) successfully deleted
W20250905 19:52:19.330192 849 leader_election.cc:343] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 76a5c85dbda140d882a5a63a13aaabd7
I20250905 19:52:19.330353 849 leader_election.cc:304] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.330989 1413 raft_consensus.cc:2747] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.331854 1491 raft_consensus.cc:513] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.332947 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c8a8848304491bbd4a7971f96322f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:19.333470 848 leader_election.cc:343] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.333853 1491 leader_election.cc:290] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.338694 1185 ts_tablet_manager.cc:1918] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.338846 1185 log.cc:1199] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/59faa6716fed4ea6b563df5f024b2058
I20250905 19:52:19.338816 1170 raft_consensus.cc:357] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.339370 1170 raft_consensus.cc:383] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.339464 1170 raft_consensus.cc:738] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.339700 1170 consensus_queue.cc:260] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.340605 1170 ts_tablet_manager.cc:1428] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.032s user 0.002s sys 0.000s
I20250905 19:52:19.340623 1185 ts_tablet_manager.cc:1939] T 59faa6716fed4ea6b563df5f024b2058 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.341199 1170 tablet_bootstrap.cc:492] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:19.341536 1185 tablet_replica.cc:331] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.341739 1185 raft_consensus.cc:2241] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.341722 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "607c8a8848304491bbd4a7971f96322f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:19.341912 1185 raft_consensus.cc:2270] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.341939 1052 raft_consensus.cc:3058] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.342478 1185 ts_tablet_manager.cc:1905] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.343005 1170 tablet_bootstrap.cc:654] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.346179 1184 tablet_replica.cc:331] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.346186 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 59faa6716fed4ea6b563df5f024b2058 (table default.Table_37d80cf1db464cff9b77c0aff7a595c2 [id=f5891226b675435f81cbdca3c8b4f43e]) successfully deleted
I20250905 19:52:19.346395 1184 raft_consensus.cc:2241] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.346989 1184 raft_consensus.cc:2270] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.347599 1184 ts_tablet_manager.cc:1905] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.350101 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_90a255f85d934941be4b07b8d27bdd8b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.350780 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_90a255f85d934941be4b07b8d27bdd8b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:19.351369 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6f3473785fff45d692bd50b1015ffe39 consensus state has no leader
I20250905 19:52:19.356679 1052 raft_consensus.cc:2466] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:19.357326 849 leader_election.cc:304] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.357818 1491 raft_consensus.cc:2802] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:19.357954 1491 raft_consensus.cc:695] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:19.358208 1491 consensus_queue.cc:237] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.358701 1169 tablet_bootstrap.cc:654] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.381131 1185 ts_tablet_manager.cc:1918] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.381279 1185 log.cc:1199] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/f8d3cbdda0c94e139259d90919dc2f95
I20250905 19:52:19.381263 774 raft_consensus.cc:3058] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.381678 1185 ts_tablet_manager.cc:1939] T f8d3cbdda0c94e139259d90919dc2f95 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.384618 1184 ts_tablet_manager.cc:1918] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.384784 1184 log.cc:1199] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:19.385221 1184 ts_tablet_manager.cc:1939] T d480531ba94f4b0f9fe40e88fe91512e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.396597 1184 tablet_replica.cc:331] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.397713 1169 tablet_bootstrap.cc:492] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.397933 1169 ts_tablet_manager.cc:1397] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.088s user 0.004s sys 0.000s
I20250905 19:52:19.398908 1169 raft_consensus.cc:357] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.399158 1169 raft_consensus.cc:383] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.399245 1169 raft_consensus.cc:738] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.399515 1169 consensus_queue.cc:260] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.408280 1169 ts_tablet_manager.cc:1428] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250905 19:52:19.409494 1508 tablet.cc:1722] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:19.409838 1169 tablet_bootstrap.cc:492] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.412686 1169 tablet_bootstrap.cc:654] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.414630 1236 raft_consensus.cc:491] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.414850 1236 raft_consensus.cc:513] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.415592 1236 leader_election.cc:290] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.416683 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a41dbd8782460cb0d3c5f8bd59354f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.416890 776 raft_consensus.cc:3058] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:19.418150 986 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.419111 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a41dbd8782460cb0d3c5f8bd59354f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.420073 984 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.420225 984 leader_election.cc:304] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.420701 1236 raft_consensus.cc:2747] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.422513 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d480531ba94f4b0f9fe40e88fe91512e (table default.Table_c7b1f5de2f2f4657971e14388d57ef5a [id=d92e1dc4017f4f8cbfdeeb40900f136c]) successfully deleted
I20250905 19:52:19.424831 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_90d6fa79cea5482eab5d6b62b5d7181b" } modify_external_catalogs: true
I20250905 19:52:19.425094 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 2683a36998674f739b41d2a57339954c
I20250905 19:52:19.425169 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:19.425874 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 091ef981d6b54756830927ea111e78aa
I20250905 19:52:19.427209 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.428006 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet f8d3cbdda0c94e139259d90919dc2f95 (table default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
W20250905 19:52:19.428596 441 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 43111a3512fb4fc0bdf4ee245b16695b because the tablet was not found. No further retry: Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
W20250905 19:52:19.429046 442 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 45524.9us
I20250905 19:52:19.429150 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 42f3c1efb94e446fb0128b6b412821e2
I20250905 19:52:19.424366 447 catalog_manager.cc:5582] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:19.429884 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:19.430155 442 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a884eca0e54a44dc920114e61a199689 because the tablet was not found. No further retry: Not found: Tablet not found: a884eca0e54a44dc920114e61a199689
W20250905 19:52:19.430481 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 98e5ca7bd3e84916a11e01afbf48171d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98e5ca7bd3e84916a11e01afbf48171d already in progress: creating tablet
I20250905 19:52:19.430601 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.430692 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.431252 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_90d6fa79cea5482eab5d6b62b5d7181b" } modify_external_catalogs: true
W20250905 19:52:19.432520 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 2683a36998674f739b41d2a57339954c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2683a36998674f739b41d2a57339954c already in progress: creating tablet
I20250905 19:52:19.432885 1184 raft_consensus.cc:2241] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.433001 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.433112 1184 raft_consensus.cc:2270] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.433236 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.433248 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.433461 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.433534 447 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:19.433717 1184 ts_tablet_manager.cc:1905] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.436105 447 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_3f876cdc8a6e4380a81a2b4d49cfb138 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 1 got 0
I20250905 19:52:19.436407 447 catalog_manager.cc:5582] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
W20250905 19:52:19.437770 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 42f3c1efb94e446fb0128b6b412821e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42f3c1efb94e446fb0128b6b412821e2 already in progress: creating tablet
I20250905 19:52:19.438119 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_450fbb5afb874baba3ecfd350509f6f0" } modify_external_catalogs: true
I20250905 19:52:19.439584 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.439687 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.439867 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:19.444064 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 091ef981d6b54756830927ea111e78aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 091ef981d6b54756830927ea111e78aa already in progress: creating tablet
I20250905 19:52:19.444641 450 catalog_manager.cc:6652] ReplaceTablet: tablet 2683a36998674f739b41d2a57339954c deleted and replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438
I20250905 19:52:19.445055 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c9fd3a3508c14b0b8d403de08fde0b82" } modify_external_catalogs: true
I20250905 19:52:19.445111 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.445515 451 catalog_manager.cc:6652] ReplaceTablet: tablet 091ef981d6b54756830927ea111e78aa deleted and replaced by tablet dc95a83ca872495894ef439cdda4e364
I20250905 19:52:19.446278 1169 tablet_bootstrap.cc:492] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.446209 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c9fd3a3508c14b0b8d403de08fde0b82" } modify_external_catalogs: true
I20250905 19:52:19.446456 1169 ts_tablet_manager.cc:1397] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.037s user 0.005s sys 0.000s
I20250905 19:52:19.447081 454 catalog_manager.cc:6652] ReplaceTablet: tablet 42f3c1efb94e446fb0128b6b412821e2 deleted and replaced by tablet 239539ac19124de192cadae494720a2b
I20250905 19:52:19.447434 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b74cc30aa834421da92e129c76067622" } new_table_name: "default.Table_05541d16561646a2b97139e3503cf120" modify_external_catalogs: true
I20250905 19:52:19.447407 1169 raft_consensus.cc:357] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.447610 1169 raft_consensus.cc:383] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.447705 1169 raft_consensus.cc:738] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.448024 1169 consensus_queue.cc:260] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.448681 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_450fbb5afb874baba3ecfd350509f6f0" } modify_external_catalogs: true
W20250905 19:52:19.451200 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 98e5ca7bd3e84916a11e01afbf48171d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98e5ca7bd3e84916a11e01afbf48171d already in progress: creating tablet
W20250905 19:52:19.451548 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 42f3c1efb94e446fb0128b6b412821e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42f3c1efb94e446fb0128b6b412821e2 already in progress: creating tablet
W20250905 19:52:19.451817 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 091ef981d6b54756830927ea111e78aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 091ef981d6b54756830927ea111e78aa already in progress: creating tablet
I20250905 19:52:19.452172 446 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]). Expected version 1 got 0
I20250905 19:52:19.452589 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e88b4d40afe0429daf95a8650aee6126" } new_table_name: "default.Table_c7c4cd0bb1b14ead83f2942518f4c30f" modify_external_catalogs: true
I20250905 19:52:19.453982 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9093f45e0bc746fba6bea50cc943e765"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.454665 1169 ts_tablet_manager.cc:1428] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
W20250905 19:52:19.454674 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9093f45e0bc746fba6bea50cc943e765 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:19.457296 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2683a36998674f739b41d2a57339954c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2683a36998674f739b41d2a57339954c already in progress: creating tablet
I20250905 19:52:19.460721 1169 tablet_bootstrap.cc:492] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.465152 1169 tablet_bootstrap.cc:654] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.466325 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e65ca281d8804277a83ab7bfc2ab9cca" } modify_external_catalogs: true
I20250905 19:52:19.466549 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e65ca281d8804277a83ab7bfc2ab9cca" } modify_external_catalogs: true
I20250905 19:52:19.476258 1169 tablet_bootstrap.cc:492] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.476478 1169 ts_tablet_manager.cc:1397] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250905 19:52:19.477465 1169 raft_consensus.cc:357] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.477737 1169 raft_consensus.cc:383] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.477835 1169 raft_consensus.cc:738] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.478185 1169 consensus_queue.cc:260] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.484680 1169 ts_tablet_manager.cc:1428] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
W20250905 19:52:19.485634 1537 log.cc:927] Time spent T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Append to log took a long time: real 0.087s user 0.000s sys 0.000s
I20250905 19:52:19.490654 1494 consensus_queue.cc:1035] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:19.491413 1169 tablet_bootstrap.cc:492] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.491634 1170 tablet_bootstrap.cc:492] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.491787 1170 ts_tablet_manager.cc:1397] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.151s user 0.007s sys 0.021s
I20250905 19:52:19.492581 1170 raft_consensus.cc:357] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.492734 1170 raft_consensus.cc:383] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.492805 1170 raft_consensus.cc:738] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.492988 1170 consensus_queue.cc:260] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.493459 1170 ts_tablet_manager.cc:1428] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:19.493600 1169 tablet_bootstrap.cc:654] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.493767 1170 tablet_bootstrap.cc:492] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:19.494303 1184 ts_tablet_manager.cc:1918] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.494452 1184 log.cc:1199] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/77cbdda0e2d641bd8092f00925bb9a9c
I20250905 19:52:19.494850 1184 ts_tablet_manager.cc:1939] T 77cbdda0e2d641bd8092f00925bb9a9c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.495337 1170 tablet_bootstrap.cc:654] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.501387 1236 raft_consensus.cc:491] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.501592 1236 raft_consensus.cc:513] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.502346 1236 leader_election.cc:290] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.503206 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36f966bf512d4cbe9ae406386db31533" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:19.504947 986 leader_election.cc:343] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.505193 1494 raft_consensus.cc:491] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.505388 1494 raft_consensus.cc:513] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.506208 1494 leader_election.cc:290] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:19.507340 849 leader_election.cc:343] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9d6daaa0fe43414299c032d1f4d9bb6f
I20250905 19:52:19.508742 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9d6daaa0fe43414299c032d1f4d9bb6f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.509558 1184 tablet_replica.cc:331] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
W20250905 19:52:19.509555 848 leader_election.cc:343] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.509721 848 leader_election.cc:304] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.509822 1184 raft_consensus.cc:2241] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.510030 1184 raft_consensus.cc:2270] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.510290 1494 raft_consensus.cc:2747] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.510672 1184 ts_tablet_manager.cc:1905] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.508774 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36f966bf512d4cbe9ae406386db31533" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.511356 984 leader_election.cc:343] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.511499 984 leader_election.cc:304] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.512046 1236 raft_consensus.cc:2747] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.513681 1171 tablet_bootstrap.cc:492] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.513880 1171 ts_tablet_manager.cc:1397] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.205s user 0.037s sys 0.046s
I20250905 19:52:19.515164 1171 raft_consensus.cc:357] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.515450 1171 raft_consensus.cc:383] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.515615 1171 raft_consensus.cc:738] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:19.516005 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e89d23c4cb64f568b25f92a278b9d5e consensus state has no leader
I20250905 19:52:19.515941 1171 consensus_queue.cc:260] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.516471 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_247bc51043654b48925883ad1c002451"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.516948 1171 ts_tablet_manager.cc:1428] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
W20250905 19:52:19.517163 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_247bc51043654b48925883ad1c002451 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.517258 1171 tablet_bootstrap.cc:492] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.519022 1171 tablet_bootstrap.cc:654] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.520714 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]). Expected version 1 got 0
I20250905 19:52:19.520942 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:19.521209 455 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:19.522326 1171 tablet_bootstrap.cc:492] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.522490 1171 ts_tablet_manager.cc:1397] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20250905 19:52:19.523192 1171 raft_consensus.cc:357] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.525614 1171 raft_consensus.cc:383] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.525831 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d20e269bf2224fe49f701c4777c42632
I20250905 19:52:19.525980 1171 raft_consensus.cc:738] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.526434 1171 consensus_queue.cc:260] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.526643 448 master_service.cc:948] ReplaceTablet: received request to replace tablet a5af8232d31240ff855b79dcea29396b from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:19.528136 1171 ts_tablet_manager.cc:1428] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:19.528426 1171 tablet_bootstrap.cc:492] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.528843 888 tablet_service.cc:1515] Processing DeleteTablet for tablet d20e269bf2224fe49f701c4777c42632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.529111 1185 tablet_replica.cc:331] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.529325 1185 raft_consensus.cc:2241] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.529541 1185 raft_consensus.cc:2270] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.529608 746 tablet_service.cc:1515] Processing DeleteTablet for tablet d20e269bf2224fe49f701c4777c42632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.529860 1186 tablet_replica.cc:331] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:19.530050 1186 raft_consensus.cc:2241] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.530113 1185 ts_tablet_manager.cc:1905] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.530230 1186 raft_consensus.cc:2270] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.530961 1169 tablet_bootstrap.cc:492] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.531109 1169 ts_tablet_manager.cc:1397] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20250905 19:52:19.531901 1169 raft_consensus.cc:357] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.532130 1169 raft_consensus.cc:383] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.533306 1169 raft_consensus.cc:738] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.533694 1186 ts_tablet_manager.cc:1905] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.533911 1169 consensus_queue.cc:260] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:19.534586 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e consensus state has no leader
I20250905 19:52:19.534976 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_fe588d1872d847a7b9cd61b07c6f7abb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.535497 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe588d1872d847a7b9cd61b07c6f7abb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.537858 1185 ts_tablet_manager.cc:1918] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:19.537945 1169 ts_tablet_manager.cc:1428] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 19:52:19.538017 1185 log.cc:1199] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d20e269bf2224fe49f701c4777c42632
I20250905 19:52:19.538450 1185 ts_tablet_manager.cc:1939] T d20e269bf2224fe49f701c4777c42632 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.539125 1169 tablet_bootstrap.cc:492] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.539518 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]) successfully deleted
I20250905 19:52:19.540933 1171 tablet_bootstrap.cc:654] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.541539 1186 ts_tablet_manager.cc:1918] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:19.541785 1186 log.cc:1199] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d20e269bf2224fe49f701c4777c42632
I20250905 19:52:19.542281 1186 ts_tablet_manager.cc:1939] T d20e269bf2224fe49f701c4777c42632 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:19.542950 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]) successfully deleted
I20250905 19:52:19.551434 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 2683a36998674f739b41d2a57339954c from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:19.552460 1169 tablet_bootstrap.cc:654] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:19.553159 441 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.60:34395, user_credentials={real_user=slave}} blocked reactor thread for 67034.9us
I20250905 19:52:19.553925 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet d20e269bf2224fe49f701c4777c42632 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.554765 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 77cbdda0e2d641bd8092f00925bb9a9c (table default.Table_39151a7d21ac4d49bb5728f518e3db4f [id=e0a1acd20bc340088714693c74f31d2f]) successfully deleted
I20250905 19:52:19.555650 1184 ts_tablet_manager.cc:1918] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.556566 1184 log.cc:1199] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e6f837fb7293474b8ad357f52e42e42e
I20250905 19:52:19.557904 1184 ts_tablet_manager.cc:1939] T e6f837fb7293474b8ad357f52e42e42e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.559211 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e6f837fb7293474b8ad357f52e42e42e (table default.Table_1fd86ae5263542ed92214968ec16ccf0 [id=02ab99d2ef0f499db0d3625ede6c6e6e]) successfully deleted
I20250905 19:52:19.564283 1236 raft_consensus.cc:491] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.564499 1236 raft_consensus.cc:513] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.565173 1170 tablet_bootstrap.cc:492] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.565342 1236 leader_election.cc:290] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.565369 1170 ts_tablet_manager.cc:1397] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.072s user 0.008s sys 0.002s
I20250905 19:52:19.566111 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c76077e4adf45a7adc736b0e69032eb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.566394 1170 raft_consensus.cc:357] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.566677 1170 raft_consensus.cc:383] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.566839 1170 raft_consensus.cc:738] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.567243 1170 consensus_queue.cc:260] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.567353 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c76077e4adf45a7adc736b0e69032eb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.568055 1170 ts_tablet_manager.cc:1428] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
W20250905 19:52:19.568210 984 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.568346 1170 tablet_bootstrap.cc:492] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:19.569207 986 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.569379 986 leader_election.cc:304] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.570441 1236 raft_consensus.cc:2747] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.571204 1170 tablet_bootstrap.cc:654] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:19.571316 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 98e5ca7bd3e84916a11e01afbf48171d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98e5ca7bd3e84916a11e01afbf48171d already in progress: creating tablet
W20250905 19:52:19.572207 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 091ef981d6b54756830927ea111e78aa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 091ef981d6b54756830927ea111e78aa already in progress: creating tablet
W20250905 19:52:19.573204 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 42f3c1efb94e446fb0128b6b412821e2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 42f3c1efb94e446fb0128b6b412821e2 already in progress: creating tablet
W20250905 19:52:19.574256 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 2683a36998674f739b41d2a57339954c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2683a36998674f739b41d2a57339954c already in progress: creating tablet
I20250905 19:52:19.575659 1184 tablet_replica.cc:331] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.576043 1184 raft_consensus.cc:2241] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:19.576551 1184 raft_consensus.cc:2270] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.577620 1169 tablet_bootstrap.cc:492] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.578065 1169 ts_tablet_manager.cc:1397] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.039s user 0.004s sys 0.002s
I20250905 19:52:19.579139 1169 raft_consensus.cc:357] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.579521 1169 raft_consensus.cc:383] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.579665 1169 raft_consensus.cc:738] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.580093 1169 consensus_queue.cc:260] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.583757 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 09be30768f5742cf9a4bf1e3fe86dfe8
I20250905 19:52:19.584532 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_97197831cd2f45cc909b9d7ab0caebc2" } new_table_name: "default.Table_d1234b81a24b4cdfbcfe61677e6bb957" modify_external_catalogs: true
I20250905 19:52:19.584645 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.584929 1186 tablet_replica.cc:331] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:19.585107 1186 raft_consensus.cc:2241] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.585326 1186 raft_consensus.cc:2270] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.585312 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.585985 1169 ts_tablet_manager.cc:1428] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250905 19:52:19.586228 1186 ts_tablet_manager.cc:1905] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.586338 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e304b44cbe8f4c77bae2af66f9b5920e" } new_table_name: "default.Table_30534826add54f68973738b8e5c471f0" modify_external_catalogs: true
I20250905 19:52:19.588088 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.588708 1185 tablet_replica.cc:331] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.588908 1185 raft_consensus.cc:2241] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:19.589354 1185 raft_consensus.cc:2270] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.589965 1185 ts_tablet_manager.cc:1905] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.590025 1169 tablet_bootstrap.cc:492] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.590405 1236 raft_consensus.cc:491] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.591367 1236 raft_consensus.cc:513] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.592197 1236 leader_election.cc:290] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.593032 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3602094b96c14e66b997e8035fc53c40" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.594435 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3602094b96c14e66b997e8035fc53c40" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.595196 986 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:19.595530 984 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.595681 984 leader_election.cc:304] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.596302 1236 raft_consensus.cc:2747] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.598477 1184 ts_tablet_manager.cc:1905] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.599210 1185 ts_tablet_manager.cc:1918] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:19.599491 1169 tablet_bootstrap.cc:654] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.599911 1171 tablet_bootstrap.cc:492] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.600069 1171 ts_tablet_manager.cc:1397] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.072s user 0.025s sys 0.033s
I20250905 19:52:19.600172 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a3805127dd78489e8545da7414ead904
I20250905 19:52:19.600885 1171 raft_consensus.cc:357] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.601070 1171 raft_consensus.cc:383] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.601145 1171 raft_consensus.cc:738] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.601365 1171 consensus_queue.cc:260] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.601850 1171 ts_tablet_manager.cc:1428] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:19.602972 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:19.604521 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_5f2927da18ef4c51a2b70428fb215a76"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.604626 746 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:19.605084 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5f2927da18ef4c51a2b70428fb215a76 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.606240 1186 ts_tablet_manager.cc:1918] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:19.606369 1186 log.cc:1199] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/09be30768f5742cf9a4bf1e3fe86dfe8
I20250905 19:52:19.606815 1186 ts_tablet_manager.cc:1939] T 09be30768f5742cf9a4bf1e3fe86dfe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:19.607528 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.607919 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]) successfully deleted
I20250905 19:52:19.608681 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.609257 1185 log.cc:1199] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/09be30768f5742cf9a4bf1e3fe86dfe8
W20250905 19:52:19.609701 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a3805127dd78489e8545da7414ead904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3805127dd78489e8545da7414ead904 already in progress: creating tablet
I20250905 19:52:19.609800 1185 ts_tablet_manager.cc:1939] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.610417 1171 tablet_bootstrap.cc:492] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.610803 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]) successfully deleted
I20250905 19:52:19.614641 1171 tablet_bootstrap.cc:654] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.615808 1184 ts_tablet_manager.cc:1918] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:19.616847 1184 log.cc:1199] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d20e269bf2224fe49f701c4777c42632
I20250905 19:52:19.617497 1184 ts_tablet_manager.cc:1939] T d20e269bf2224fe49f701c4777c42632 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.620419 1169 tablet_bootstrap.cc:492] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.620594 1169 ts_tablet_manager.cc:1397] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.031s user 0.011s sys 0.005s
I20250905 19:52:19.620546 447 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_4447fb37960e42dfa2c6eb8ced9c6b68 [id=99384aac4e624d6db630b249c3ba2543] alter complete (version 1)
I20250905 19:52:19.621407 1169 raft_consensus.cc:357] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.621590 1169 raft_consensus.cc:383] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.621675 1169 raft_consensus.cc:738] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.621526 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_934044a3a6e849528a30d212f3dc5fd5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.621975 1184 tablet_replica.cc:331] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.622164 1184 raft_consensus.cc:2241] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:19.622229 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_934044a3a6e849528a30d212f3dc5fd5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.622357 1184 raft_consensus.cc:2270] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.622911 1184 ts_tablet_manager.cc:1905] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.624106 1169 consensus_queue.cc:260] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.624445 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_88ff02c5dce2458784f09b64cee688f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.625078 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_88ff02c5dce2458784f09b64cee688f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.625138 1169 ts_tablet_manager.cc:1428] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:19.626962 1169 tablet_bootstrap.cc:492] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.627516 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 with delete_type TABLET_DATA_DELETED (replaced by tablet f28e403dc2f6471ebea1805fd843cfb1) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.627856 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.628521 1169 tablet_bootstrap.cc:654] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.634697 1184 ts_tablet_manager.cc:1918] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:19.634825 1184 log.cc:1199] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/09be30768f5742cf9a4bf1e3fe86dfe8
I20250905 19:52:19.635253 1184 ts_tablet_manager.cc:1939] T 09be30768f5742cf9a4bf1e3fe86dfe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.636544 1171 tablet_bootstrap.cc:492] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.636718 1171 ts_tablet_manager.cc:1397] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.005s
I20250905 19:52:19.636723 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.637005 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 1664cbe248514535953a1ef2d312f98f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:15 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.637223 753 tablet_service.cc:1515] Processing DeleteTablet for tablet f8d3cbdda0c94e139259d90919dc2f95 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.637441 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 59faa6716fed4ea6b563df5f024b2058 with delete_type TABLET_DATA_DELETED (replaced by tablet 05ec559d0f83445892f8285f9bdeb2c8) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.637517 1171 raft_consensus.cc:357] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.637729 1171 raft_consensus.cc:383] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.637801 1171 raft_consensus.cc:738] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.637935 1186 tablet_replica.cc:331] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:19.638216 1186 raft_consensus.cc:2241] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.638388 1236 raft_consensus.cc:491] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.638424 1186 raft_consensus.cc:2270] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.638561 1236 raft_consensus.cc:513] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.639560 1236 leader_election.cc:290] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.639571 1186 ts_tablet_manager.cc:1905] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.639729 1171 consensus_queue.cc:260] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.639572 1169 tablet_bootstrap.cc:492] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.640156 1169 ts_tablet_manager.cc:1397] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.000s
I20250905 19:52:19.641270 1169 raft_consensus.cc:357] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.641448 1169 raft_consensus.cc:383] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.641431 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.641523 1169 raft_consensus.cc:738] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.641738 1169 consensus_queue.cc:260] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.642339 1169 ts_tablet_manager.cc:1428] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:19.642657 1169 tablet_bootstrap.cc:492] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.644228 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.644557 1171 ts_tablet_manager.cc:1428] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250905 19:52:19.645226 1491 raft_consensus.cc:491] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.645421 1491 raft_consensus.cc:513] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:19.645597 986 leader_election.cc:343] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.646039 1491 leader_election.cc:290] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:19.646904 984 leader_election.cc:343] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.646994 1171 tablet_bootstrap.cc:492] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.647048 984 leader_election.cc:304] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.647809 1394 raft_consensus.cc:2747] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:19.648375 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 09be30768f5742cf9a4bf1e3fe86dfe8 because the tablet was not found. No further retry: Not found: Tablet not found: 09be30768f5742cf9a4bf1e3fe86dfe8
I20250905 19:52:19.648675 1171 tablet_bootstrap.cc:654] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.650210 1169 tablet_bootstrap.cc:654] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.651942 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d20e269bf2224fe49f701c4777c42632 (table default.Table_450fbb5afb874baba3ecfd350509f6f0 [id=5cdcc43b222346d481565b3d3655d540]) successfully deleted
I20250905 19:52:19.657733 1186 ts_tablet_manager.cc:1918] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.657920 1186 log.cc:1199] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/1664cbe248514535953a1ef2d312f98f
W20250905 19:52:19.658275 849 leader_election.cc:343] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 7edcf57f7753449d8a5fcdb6f4f1dc21
I20250905 19:52:19.658351 1186 ts_tablet_manager.cc:1939] T 1664cbe248514535953a1ef2d312f98f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:19.659060 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 1664cbe248514535953a1ef2d312f98f (table default.Table_6a14b2be50f942d8bfd8c47247512879 [id=353aa71a6ee64c41aa94daadd629d845]) successfully deleted
I20250905 19:52:19.659432 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.659704 1184 tablet_replica.cc:331] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
W20250905 19:52:19.659909 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a3805127dd78489e8545da7414ead904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3805127dd78489e8545da7414ead904 already in progress: creating tablet
I20250905 19:52:19.659929 1184 raft_consensus.cc:2241] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.660097 1394 raft_consensus.cc:491] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.660259 1184 raft_consensus.cc:2270] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.660234 1394 raft_consensus.cc:513] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.660351 1185 tablet_replica.cc:331] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.660665 1491 raft_consensus.cc:491] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.660837 1394 leader_election.cc:290] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.659706 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.660890 1491 raft_consensus.cc:513] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.661302 1184 ts_tablet_manager.cc:1905] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.661437 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7edcf57f7753449d8a5fcdb6f4f1dc21" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.661689 1491 leader_election.cc:290] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.660660 1185 raft_consensus.cc:2241] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:19.662097 848 leader_election.cc:343] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.662246 848 leader_election.cc:304] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.662312 1185 raft_consensus.cc:2270] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.662631 1494 raft_consensus.cc:2747] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.663626 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fac0e1f483b40ab8d66cb5a9ee808fc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:19.663846 1053 raft_consensus.cc:2466] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:19.667515 1184 ts_tablet_manager.cc:1918] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.668457 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a5af8232d31240ff855b79dcea29396b
I20250905 19:52:19.668995 448 catalog_manager.cc:6652] ReplaceTablet: tablet a5af8232d31240ff855b79dcea29396b deleted and replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2
I20250905 19:52:19.670358 1184 log.cc:1199] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:19.671115 1184 ts_tablet_manager.cc:1939] T 60a41dbd8782460cb0d3c5f8bd59354f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:19.672171 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c20eea1b024d7f93f18f0c089efc42" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.672232 849 leader_election.cc:304] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:19.672642 1494 raft_consensus.cc:2802] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:19.672768 1494 raft_consensus.cc:491] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:19.672886 1494 raft_consensus.cc:3058] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.673072 1184 tablet_replica.cc:331] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:19.673267 1184 raft_consensus.cc:2241] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.673277 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c20eea1b024d7f93f18f0c089efc42" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.673442 1184 raft_consensus.cc:2270] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.673794 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fac0e1f483b40ab8d66cb5a9ee808fc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.673969 1184 ts_tablet_manager.cc:1905] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:19.674244 848 leader_election.cc:343] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:19.675263 986 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:19.675639 984 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.675774 984 leader_election.cc:304] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.676067 1394 raft_consensus.cc:491] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.676211 1394 raft_consensus.cc:513] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.662113 1491 raft_consensus.cc:491] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.677127 1491 raft_consensus.cc:513] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.677243 1394 leader_election.cc:290] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.677379 1394 raft_consensus.cc:2747] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.677471 885 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:19.677836 1491 leader_election.cc:290] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:19.679075 849 leader_election.cc:343] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: e6f837fb7293474b8ad357f52e42e42e
I20250905 19:52:19.683362 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af5748bea3bd4f51a0d6c28a1d9044a0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:19.684907 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6f837fb7293474b8ad357f52e42e42e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.685097 1494 raft_consensus.cc:513] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:19.685667 986 leader_election.cc:343] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:19.686017 848 leader_election.cc:343] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.686156 848 leader_election.cc:304] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.686861 1494 leader_election.cc:290] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.687078 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fac0e1f483b40ab8d66cb5a9ee808fc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:19.687158 1494 raft_consensus.cc:2747] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.687323 1053 raft_consensus.cc:3058] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.688342 744 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:19.690433 1053 raft_consensus.cc:2466] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:19.691105 849 leader_election.cc:304] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:19.691470 1184 ts_tablet_manager.cc:1918] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.691664 1494 raft_consensus.cc:2802] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:19.691924 1494 raft_consensus.cc:695] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:19.692337 1494 consensus_queue.cc:237] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.692507 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af5748bea3bd4f51a0d6c28a1d9044a0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.693159 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fac0e1f483b40ab8d66cb5a9ee808fc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:19.694924 848 leader_election.cc:343] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.699373 1184 log.cc:1199] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/3602094b96c14e66b997e8035fc53c40
I20250905 19:52:19.699774 1170 tablet_bootstrap.cc:492] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:19.699626 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a21b18f7481c4713823fadf4b164b7bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.694533 984 leader_election.cc:343] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.700111 984 leader_election.cc:304] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.700196 1170 ts_tablet_manager.cc:1397] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.132s user 0.004s sys 0.000s
I20250905 19:52:19.700263 1184 ts_tablet_manager.cc:1939] T 3602094b96c14e66b997e8035fc53c40 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:19.700318 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a21b18f7481c4713823fadf4b164b7bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.700675 1236 raft_consensus.cc:2747] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.701303 1170 raft_consensus.cc:357] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.701583 1170 raft_consensus.cc:383] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.701758 1170 raft_consensus.cc:738] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.702095 1170 consensus_queue.cc:260] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.702726 776 raft_consensus.cc:3058] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.706377 1236 raft_consensus.cc:491] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.706604 1236 raft_consensus.cc:513] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.707418 1236 leader_election.cc:290] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.708247 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68f2ccd8a7a64b89b44f2d6767ec3466" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:19.709776 986 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.712245 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68f2ccd8a7a64b89b44f2d6767ec3466" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.712913 984 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.713079 984 leader_election.cc:304] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.713657 1236 raft_consensus.cc:2747] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.726994 1186 tablet_replica.cc:331] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:19.727988 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:19.727337 1508 tablet.cc:1722] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:19.728796 1170 ts_tablet_manager.cc:1428] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.028s user 0.002s sys 0.000s
I20250905 19:52:19.729673 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 09be30768f5742cf9a4bf1e3fe86dfe8 (table default.Table_c9fd3a3508c14b0b8d403de08fde0b82 [id=6b4e399e326f47ad8c3ca4d5e33538d4]) successfully deleted
W20250905 19:52:19.729979 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a3805127dd78489e8545da7414ead904 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a3805127dd78489e8545da7414ead904 already in progress: creating tablet
I20250905 19:52:19.730952 1170 tablet_bootstrap.cc:492] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:19.731124 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 60a41dbd8782460cb0d3c5f8bd59354f (table default.Table_cddec42a8c1c415ba27927b52d055a08 [id=c60ff09c1e55439cbac9002d52e8d546]) successfully deleted
I20250905 19:52:19.731401 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 3602094b96c14e66b997e8035fc53c40 (table default.Table_ca7a00374cab4386a886afd1d3a90342 [id=f7d8cd0a6972466a997a8fdd26bccf46]) successfully deleted
W20250905 19:52:19.732157 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a5af8232d31240ff855b79dcea29396b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5af8232d31240ff855b79dcea29396b already in progress: creating tablet
I20250905 19:52:19.735030 1170 tablet_bootstrap.cc:654] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.735474 1186 raft_consensus.cc:2241] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.735759 1186 raft_consensus.cc:2270] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:19.736796 1186 ts_tablet_manager.cc:1905] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:19.743232 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54628965b82e470d811eff95ffc56c5f consensus state has no leader
W20250905 19:52:19.745872 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3315d3e9633e42ea8d7a6b5934cb0438 has no consensus state
I20250905 19:52:19.746276 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a19dd25caac94580b689031eda61e7e4" } new_table_name: "default.Table_2df584b8fa57457bb948319fd6619a7a" modify_external_catalogs: true
I20250905 19:52:19.757352 1185 ts_tablet_manager.cc:1905] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.765098 1171 tablet_bootstrap.cc:492] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.765275 1171 ts_tablet_manager.cc:1397] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.118s user 0.025s sys 0.023s
I20250905 19:52:19.766085 1171 raft_consensus.cc:357] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.766268 1171 raft_consensus.cc:383] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.766351 1171 raft_consensus.cc:738] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.766664 1171 consensus_queue.cc:260] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.768250 1169 tablet_bootstrap.cc:492] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.768404 1169 ts_tablet_manager.cc:1397] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.126s user 0.014s sys 0.035s
I20250905 19:52:19.769212 1169 raft_consensus.cc:357] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.769397 1169 raft_consensus.cc:383] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.769570 1169 raft_consensus.cc:738] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.769860 1169 consensus_queue.cc:260] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.770833 1169 ts_tablet_manager.cc:1428] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:19.771373 1169 tablet_bootstrap.cc:492] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.773097 1171 ts_tablet_manager.cc:1428] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:19.773929 1171 tablet_bootstrap.cc:492] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.774673 1169 tablet_bootstrap.cc:654] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.776873 1185 ts_tablet_manager.cc:1918] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.777634 1171 tablet_bootstrap.cc:654] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.780512 1169 tablet_bootstrap.cc:492] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.780716 1169 ts_tablet_manager.cc:1397] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20250905 19:52:19.781561 1185 log.cc:1199] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/76a5c85dbda140d882a5a63a13aaabd7
I20250905 19:52:19.781610 1169 raft_consensus.cc:357] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.781836 1169 raft_consensus.cc:383] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.781920 1169 raft_consensus.cc:738] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.782192 1169 consensus_queue.cc:260] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.782336 1185 ts_tablet_manager.cc:1939] T 76a5c85dbda140d882a5a63a13aaabd7 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.783438 1169 ts_tablet_manager.cc:1428] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250905 19:52:19.784492 1185 tablet_replica.cc:331] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:19.785636 1169 tablet_bootstrap.cc:492] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.786600 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:19.786976 1185 raft_consensus.cc:2241] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:19.787186 1185 raft_consensus.cc:2270] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:19.787333 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.787734 1185 ts_tablet_manager.cc:1905] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:19.788072 1169 tablet_bootstrap.cc:654] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.789247 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_2af22804e18e430ea738862d0bade226"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.789956 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2af22804e18e430ea738862d0bade226 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.791457 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 76a5c85dbda140d882a5a63a13aaabd7 (table default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]) successfully deleted
I20250905 19:52:19.797142 1236 raft_consensus.cc:491] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.797380 1236 raft_consensus.cc:513] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.798213 1236 leader_election.cc:290] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.799319 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f581403ec924fb0948400df41ed8aab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:19.799806 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f581403ec924fb0948400df41ed8aab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:19.800274 984 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.800601 1185 ts_tablet_manager.cc:1918] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:19.800715 1185 log.cc:1199] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9d6daaa0fe43414299c032d1f4d9bb6f
W20250905 19:52:19.800818 986 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.800931 986 leader_election.cc:304] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.801110 1185 ts_tablet_manager.cc:1939] T 9d6daaa0fe43414299c032d1f4d9bb6f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:19.801570 1236 raft_consensus.cc:2747] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.802150 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9d6daaa0fe43414299c032d1f4d9bb6f (table default.Table_363859b13b284790a71ced21040081e4 [id=3d169e77e70f4a059cdf41f2f3a70171]) successfully deleted
I20250905 19:52:19.802759 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7df10c45b04b4316ad24e8e2d141de9b" } modify_external_catalogs: true
I20250905 19:52:19.803005 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7df10c45b04b4316ad24e8e2d141de9b" } modify_external_catalogs: true
W20250905 19:52:19.806334 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a5af8232d31240ff855b79dcea29396b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5af8232d31240ff855b79dcea29396b already in progress: creating tablet
I20250905 19:52:19.807053 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a00174b61598433595c06dcf7aadbe8f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:19.807756 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a00174b61598433595c06dcf7aadbe8f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:19.813290 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 578a6b9224cc4d8585d6c4b241762e1b from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:19.816442 744 tablet_service.cc:1468] Processing CreateTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff (DEFAULT_TABLE table=default.Table_1c6571236cd7434d98aec149d06dc6b8 [id=6709b02c192c4ff88715b1f3b886b222]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.817039 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.821285 449 catalog_manager.cc:5582] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:19.821842 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_3f876cdc8a6e4380a81a2b4d49cfb138 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 1 got 0
I20250905 19:52:19.822440 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]). Expected version 2 got 0
I20250905 19:52:19.825410 885 tablet_service.cc:1468] Processing CreateTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff (DEFAULT_TABLE table=default.Table_1c6571236cd7434d98aec149d06dc6b8 [id=6709b02c192c4ff88715b1f3b886b222]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.825883 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.829445 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff (DEFAULT_TABLE table=default.Table_1c6571236cd7434d98aec149d06dc6b8 [id=6709b02c192c4ff88715b1f3b886b222]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.829559 753 tablet_service.cc:1468] Processing CreateTablet for tablet efceaf3c7bda44ddabd5d38130af3dd2 (DEFAULT_TABLE table=default.Table_8e27522da15a4d8ebb55104bcbe5fc5f [id=e266d1ebb88f4e2e9572f6c3ceaa8530]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.829912 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.830075 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efceaf3c7bda44ddabd5d38130af3dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.833190 888 tablet_service.cc:1468] Processing CreateTablet for tablet efceaf3c7bda44ddabd5d38130af3dd2 (DEFAULT_TABLE table=default.Table_8e27522da15a4d8ebb55104bcbe5fc5f [id=e266d1ebb88f4e2e9572f6c3ceaa8530]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.833691 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efceaf3c7bda44ddabd5d38130af3dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.841189 1026 tablet_service.cc:1468] Processing CreateTablet for tablet efceaf3c7bda44ddabd5d38130af3dd2 (DEFAULT_TABLE table=default.Table_8e27522da15a4d8ebb55104bcbe5fc5f [id=e266d1ebb88f4e2e9572f6c3ceaa8530]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.841219 739 tablet_service.cc:1468] Processing CreateTablet for tablet 8503f6918ffb44efa36214633c93e31d (DEFAULT_TABLE table=default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.841643 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8503f6918ffb44efa36214633c93e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.841697 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efceaf3c7bda44ddabd5d38130af3dd2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.861366 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8503f6918ffb44efa36214633c93e31d (DEFAULT_TABLE table=default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.861852 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8503f6918ffb44efa36214633c93e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.862174 746 tablet_service.cc:1468] Processing CreateTablet for tablet 5b0d4be788f841c2860f5dd65275b623 (DEFAULT_TABLE table=default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.862555 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0d4be788f841c2860f5dd65275b623. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.869236 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 8503f6918ffb44efa36214633c93e31d (DEFAULT_TABLE table=default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.869961 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8503f6918ffb44efa36214633c93e31d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.877729 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 5b0d4be788f841c2860f5dd65275b623 (DEFAULT_TABLE table=default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.878121 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0d4be788f841c2860f5dd65275b623. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.879881 1030 tablet_service.cc:1468] Processing CreateTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 (DEFAULT_TABLE table=default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 [id=1e2786d48c8d49cfa9cbca17c164b7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.880352 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd793bce71e344f18bb177a53ab1bfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.882279 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3587780819704008b7f4ee6abc714db6 (DEFAULT_TABLE table=default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.882689 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3587780819704008b7f4ee6abc714db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.884346 1028 tablet_service.cc:1468] Processing CreateTablet for tablet e58bbb6575fd4437b242032a281914ea (DEFAULT_TABLE table=default.Table_7fa399abb0ac4fbeaeb856e415850864 [id=406ae40a142e4b6aa89bd34aa04df8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.884749 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58bbb6575fd4437b242032a281914ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.887384 739 tablet_service.cc:1468] Processing CreateTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 (DEFAULT_TABLE table=default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 [id=1e2786d48c8d49cfa9cbca17c164b7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.887758 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd793bce71e344f18bb177a53ab1bfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.893458 753 tablet_service.cc:1468] Processing CreateTablet for tablet e58bbb6575fd4437b242032a281914ea (DEFAULT_TABLE table=default.Table_7fa399abb0ac4fbeaeb856e415850864 [id=406ae40a142e4b6aa89bd34aa04df8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.893992 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58bbb6575fd4437b242032a281914ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.894942 889 tablet_service.cc:1468] Processing CreateTablet for tablet e58bbb6575fd4437b242032a281914ea (DEFAULT_TABLE table=default.Table_7fa399abb0ac4fbeaeb856e415850864 [id=406ae40a142e4b6aa89bd34aa04df8e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.895399 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e58bbb6575fd4437b242032a281914ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.899024 888 tablet_service.cc:1468] Processing CreateTablet for tablet 5b0d4be788f841c2860f5dd65275b623 (DEFAULT_TABLE table=default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.899535 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b0d4be788f841c2860f5dd65275b623. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.904181 885 tablet_service.cc:1468] Processing CreateTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 (DEFAULT_TABLE table=default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 [id=1e2786d48c8d49cfa9cbca17c164b7eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.904691 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd793bce71e344f18bb177a53ab1bfc1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.916386 886 tablet_service.cc:1468] Processing CreateTablet for tablet d296428b70f9411190b6c408d81639f2 (DEFAULT_TABLE table=default.Table_c5d8d96a9bb34b5ea9b36388f41faf80 [id=80e5ee2179c744f7beccf76fc5ceb396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.916849 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d296428b70f9411190b6c408d81639f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.917801 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 3587780819704008b7f4ee6abc714db6 (DEFAULT_TABLE table=default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.918213 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3587780819704008b7f4ee6abc714db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.920286 879 tablet_service.cc:1468] Processing CreateTablet for tablet fb72fe6049894527917e1481bcb7b03a (DEFAULT_TABLE table=default.Table_191f4f9b06184a8a8429cce5802c520e [id=a9604dfc1b44424f97da39857b13ae91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.920661 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb72fe6049894527917e1481bcb7b03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.923851 1236 raft_consensus.cc:491] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.924139 1236 raft_consensus.cc:513] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.925091 1236 leader_election.cc:290] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:19.926952 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:19.927800 986 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.931228 1028 tablet_service.cc:1468] Processing CreateTablet for tablet d296428b70f9411190b6c408d81639f2 (DEFAULT_TABLE table=default.Table_c5d8d96a9bb34b5ea9b36388f41faf80 [id=80e5ee2179c744f7beccf76fc5ceb396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.931680 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d296428b70f9411190b6c408d81639f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.936605 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3587780819704008b7f4ee6abc714db6 (DEFAULT_TABLE table=default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.937050 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3587780819704008b7f4ee6abc714db6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.937503 1169 tablet_bootstrap.cc:492] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.937656 1169 ts_tablet_manager.cc:1397] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.152s user 0.020s sys 0.042s
I20250905 19:52:19.938454 1169 raft_consensus.cc:357] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.938632 1169 raft_consensus.cc:383] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.938730 1169 raft_consensus.cc:738] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.938958 1169 consensus_queue.cc:260] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.939680 1169 ts_tablet_manager.cc:1428] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:19.940155 1169 tablet_bootstrap.cc:492] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.943265 1171 tablet_bootstrap.cc:492] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.943434 1171 ts_tablet_manager.cc:1397] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.170s user 0.030s sys 0.087s
I20250905 19:52:19.944185 1171 raft_consensus.cc:357] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.944384 1171 raft_consensus.cc:383] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.944463 1171 raft_consensus.cc:738] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.944741 1171 consensus_queue.cc:260] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.945868 743 tablet_service.cc:1468] Processing CreateTablet for tablet fb72fe6049894527917e1481bcb7b03a (DEFAULT_TABLE table=default.Table_191f4f9b06184a8a8429cce5802c520e [id=a9604dfc1b44424f97da39857b13ae91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.946323 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb72fe6049894527917e1481bcb7b03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.947185 1171 ts_tablet_manager.cc:1428] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:19.947484 1171 tablet_bootstrap.cc:492] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.950950 1171 tablet_bootstrap.cc:654] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.952884 1169 tablet_bootstrap.cc:654] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.953630 744 tablet_service.cc:1468] Processing CreateTablet for tablet d296428b70f9411190b6c408d81639f2 (DEFAULT_TABLE table=default.Table_c5d8d96a9bb34b5ea9b36388f41faf80 [id=80e5ee2179c744f7beccf76fc5ceb396]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:19.954219 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.954922 984 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.955080 984 leader_election.cc:304] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.955634 1394 raft_consensus.cc:2747] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:19.958185 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d296428b70f9411190b6c408d81639f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:19.965541 1171 tablet_bootstrap.cc:492] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:19.965705 1171 ts_tablet_manager.cc:1397] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.005s
I20250905 19:52:19.966454 1171 raft_consensus.cc:357] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.966640 1171 raft_consensus.cc:383] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.966833 1171 raft_consensus.cc:738] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.967063 1171 consensus_queue.cc:260] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.969528 1171 ts_tablet_manager.cc:1428] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:19.972455 1171 tablet_bootstrap.cc:492] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:19.974566 1494 raft_consensus.cc:491] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:19.974723 1494 raft_consensus.cc:513] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.975337 1494 leader_election.cc:290] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:19.976627 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59c13a5b598a4597931ddf5d476aec85" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:19.977173 848 leader_election.cc:343] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.978142 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59c13a5b598a4597931ddf5d476aec85" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:19.978221 1171 tablet_bootstrap.cc:654] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.978356 1053 raft_consensus.cc:2466] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:19.978925 849 leader_election.cc:304] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.979568 1494 raft_consensus.cc:2802] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:19.979709 1494 raft_consensus.cc:491] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:19.979835 1494 raft_consensus.cc:3058] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.983652 1494 raft_consensus.cc:513] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.985013 1169 tablet_bootstrap.cc:492] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:19.985199 1169 ts_tablet_manager.cc:1397] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.045s user 0.006s sys 0.002s
I20250905 19:52:19.985354 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59c13a5b598a4597931ddf5d476aec85" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:19.985960 848 leader_election.cc:343] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:19.986034 1169 raft_consensus.cc:357] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.986236 1169 raft_consensus.cc:383] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:19.986315 1169 raft_consensus.cc:738] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:19.986562 1169 consensus_queue.cc:260] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.987216 1169 ts_tablet_manager.cc:1428] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:19.987532 1169 tablet_bootstrap.cc:492] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:19.987615 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "59c13a5b598a4597931ddf5d476aec85" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:19.988318 1053 raft_consensus.cc:3058] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:19.989914 1169 tablet_bootstrap.cc:654] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:19.991519 1053 raft_consensus.cc:2466] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:19.992298 849 leader_election.cc:304] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:19.992782 1491 raft_consensus.cc:2802] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:19.992929 1491 raft_consensus.cc:695] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:19.993271 1491 consensus_queue.cc:237] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:19.987835 1494 leader_election.cc:290] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.000833 1236 raft_consensus.cc:491] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.001055 1236 raft_consensus.cc:513] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.001847 1236 leader_election.cc:290] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.002642 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:20.003721 986 leader_election.cc:343] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.004284 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.004622 1169 tablet_bootstrap.cc:492] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.004781 1169 ts_tablet_manager.cc:1397] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.005s
W20250905 19:52:20.004848 984 leader_election.cc:343] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.004956 984 leader_election.cc:304] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.005885 1236 raft_consensus.cc:2747] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.006161 1169 raft_consensus.cc:357] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.006377 1169 raft_consensus.cc:383] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.006466 1169 raft_consensus.cc:738] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.006736 1169 consensus_queue.cc:260] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.008332 1169 ts_tablet_manager.cc:1428] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250905 19:52:20.008896 1169 tablet_bootstrap.cc:492] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.012317 1169 tablet_bootstrap.cc:654] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.017176 1170 tablet_bootstrap.cc:492] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:20.017411 1170 ts_tablet_manager.cc:1397] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.287s user 0.020s sys 0.031s
I20250905 19:52:20.018430 1170 raft_consensus.cc:357] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.018685 1170 raft_consensus.cc:383] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.018785 1170 raft_consensus.cc:738] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.019196 1170 consensus_queue.cc:260] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.022852 1170 ts_tablet_manager.cc:1428] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.002s
I20250905 19:52:20.028612 1170 tablet_bootstrap.cc:492] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:20.036689 1491 consensus_queue.cc:1035] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:20.049028 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fb72fe6049894527917e1481bcb7b03a (DEFAULT_TABLE table=default.Table_191f4f9b06184a8a8429cce5802c520e [id=a9604dfc1b44424f97da39857b13ae91]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.049710 1026 tablet_service.cc:1468] Processing CreateTablet for tablet efb70327f55844879f89449a7116010f (DEFAULT_TABLE table=default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.049808 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fb72fe6049894527917e1481bcb7b03a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.051781 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efb70327f55844879f89449a7116010f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:20.059630 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 44070.8us
I20250905 19:52:20.066141 744 tablet_service.cc:1468] Processing CreateTablet for tablet efb70327f55844879f89449a7116010f (DEFAULT_TABLE table=default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.067585 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efb70327f55844879f89449a7116010f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:20.075484 848 consensus_peers.cc:489] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:20.078934 1169 tablet_bootstrap.cc:492] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.079770 1169 ts_tablet_manager.cc:1397] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.071s user 0.027s sys 0.007s
I20250905 19:52:20.080698 1169 raft_consensus.cc:357] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.080916 1169 raft_consensus.cc:383] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.080993 1169 raft_consensus.cc:738] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.081233 1169 consensus_queue.cc:260] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.084184 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 68f2ccd8a7a64b89b44f2d6767ec3466 from {username='slave'} at 127.0.0.1:52586
W20250905 19:52:20.084575 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 4edfa9a656554e02b6649e8430c215fb consensus state has no leader
I20250905 19:52:20.085127 1171 tablet_bootstrap.cc:492] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.085309 1171 ts_tablet_manager.cc:1397] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.113s user 0.015s sys 0.032s
I20250905 19:52:20.086107 1171 raft_consensus.cc:357] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.086311 1171 raft_consensus.cc:383] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.086390 1171 raft_consensus.cc:738] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.086656 1171 consensus_queue.cc:260] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.087040 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 262bcd1cfc994bb58e999b141b9bc7ea from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.088423 1186 ts_tablet_manager.cc:1918] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.088492 1171 ts_tablet_manager.cc:1428] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:20.088635 1186 log.cc:1199] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/f8d3cbdda0c94e139259d90919dc2f95
I20250905 19:52:20.088810 1171 tablet_bootstrap.cc:492] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.089138 1186 ts_tablet_manager.cc:1939] T f8d3cbdda0c94e139259d90919dc2f95 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:20.089988 1186 tablet_replica.cc:331] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:20.090270 1186 raft_consensus.cc:2241] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.090474 1186 raft_consensus.cc:2270] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.090580 1171 tablet_bootstrap.cc:654] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.091640 1584 raft_consensus.cc:491] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.091830 1584 raft_consensus.cc:513] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.092106 1169 ts_tablet_manager.cc:1428] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20250905 19:52:20.092622 1584 leader_election.cc:290] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.092913 1169 tablet_bootstrap.cc:492] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.094130 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet f8d3cbdda0c94e139259d90919dc2f95 (table default.Table_db38b4b80e674f46846361436677e883 [id=7a9f1417bfd04c158dbedc9ea02d6dce]) successfully deleted
W20250905 19:52:20.094936 849 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:20.095289 1169 tablet_bootstrap.cc:654] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.102592 1171 tablet_bootstrap.cc:492] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.102763 1171 ts_tablet_manager.cc:1397] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.003s
I20250905 19:52:20.103699 1171 raft_consensus.cc:357] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.103844 1171 raft_consensus.cc:383] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.103933 1171 raft_consensus.cc:738] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.104143 1171 consensus_queue.cc:260] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.104605 1171 ts_tablet_manager.cc:1428] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:20.105393 1171 tablet_bootstrap.cc:492] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.108217 1171 tablet_bootstrap.cc:654] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.109331 1419 raft_consensus.cc:491] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.109513 1419 raft_consensus.cc:513] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.110080 1169 tablet_bootstrap.cc:492] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.110176 1419 leader_election.cc:290] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.110224 1169 ts_tablet_manager.cc:1397] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.002s
I20250905 19:52:20.110313 1419 consensus_queue.cc:1035] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:20.111270 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fd280a2a9f5f4f81ad7dcd3fd2b4003b" } new_table_name: "default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a" modify_external_catalogs: true
I20250905 19:52:20.115175 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19e4edddafcd4d22a0e0ae66b0b953e0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:20.115468 1053 raft_consensus.cc:2466] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:20.093500 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6d12c7fbb84849c68274810b3fdcba04" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.116700 849 leader_election.cc:304] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:20.117007 848 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.117165 848 leader_election.cc:304] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.117216 1584 raft_consensus.cc:2802] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:20.117359 1584 raft_consensus.cc:491] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:20.117480 1584 raft_consensus.cc:3058] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.117787 1419 raft_consensus.cc:2747] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.118425 1169 raft_consensus.cc:357] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.118597 1169 raft_consensus.cc:383] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.118698 1169 raft_consensus.cc:738] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.118927 1169 consensus_queue.cc:260] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.119860 1186 ts_tablet_manager.cc:1905] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.121328 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7186efc2caef4940ade686830c963ba1" } modify_external_catalogs: true
I20250905 19:52:20.121477 1169 ts_tablet_manager.cc:1428] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:20.121467 1584 raft_consensus.cc:513] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.122699 1169 tablet_bootstrap.cc:492] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.122704 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7186efc2caef4940ade686830c963ba1" } modify_external_catalogs: true
I20250905 19:52:20.123065 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19e4edddafcd4d22a0e0ae66b0b953e0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:20.123736 848 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.124436 889 tablet_service.cc:1468] Processing CreateTablet for tablet efb70327f55844879f89449a7116010f (DEFAULT_TABLE table=default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.124651 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19e4edddafcd4d22a0e0ae66b0b953e0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:20.124878 1053 raft_consensus.cc:3058] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.124965 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet efb70327f55844879f89449a7116010f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.128388 1053 raft_consensus.cc:2466] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:20.128589 1584 leader_election.cc:290] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.129431 1169 tablet_bootstrap.cc:654] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.131807 1171 tablet_bootstrap.cc:492] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.131999 1171 ts_tablet_manager.cc:1397] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250905 19:52:20.132791 1171 raft_consensus.cc:357] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.132978 1171 raft_consensus.cc:383] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.133044 1171 raft_consensus.cc:738] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.133306 1171 consensus_queue.cc:260] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.135421 1171 ts_tablet_manager.cc:1428] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:20.136404 1171 tablet_bootstrap.cc:492] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.140089 776 raft_consensus.cc:3058] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.143764 849 leader_election.cc:304] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.145948 1584 raft_consensus.cc:2802] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:20.146739 1584 raft_consensus.cc:695] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:20.147089 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_164521ea6f9342c8b5380153cb22b888" } new_table_name: "default.Table_b42374e796034b1cbfc8659587477d3c" modify_external_catalogs: true
I20250905 19:52:20.147337 1494 consensus_queue.cc:1035] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:20.149284 1169 tablet_bootstrap.cc:492] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.149482 1169 ts_tablet_manager.cc:1397] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250905 19:52:20.150102 1584 consensus_queue.cc:237] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.147910 1171 tablet_bootstrap.cc:654] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.153815 1169 raft_consensus.cc:357] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.154034 1169 raft_consensus.cc:383] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.154134 1169 raft_consensus.cc:738] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.154362 1169 consensus_queue.cc:260] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.158449 1171 tablet_bootstrap.cc:492] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.158605 1171 ts_tablet_manager.cc:1397] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.003s
I20250905 19:52:20.160490 1171 raft_consensus.cc:357] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.160686 1171 raft_consensus.cc:383] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.160794 1171 raft_consensus.cc:738] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.161041 1171 consensus_queue.cc:260] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.161674 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 578a6b9224cc4d8585d6c4b241762e1b
I20250905 19:52:20.162547 1171 ts_tablet_manager.cc:1428] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:20.162729 451 catalog_manager.cc:6652] ReplaceTablet: tablet 578a6b9224cc4d8585d6c4b241762e1b deleted and replaced by tablet a3a50526bb604a218649b24cbf65a65e
I20250905 19:52:20.163057 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_80258bf6ac33401fa32c1e2792e1d731"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.163259 1171 tablet_bootstrap.cc:492] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:20.163565 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80258bf6ac33401fa32c1e2792e1d731 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.165078 1171 tablet_bootstrap.cc:654] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.165170 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:20.167778 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 578a6b9224cc4d8585d6c4b241762e1b already in progress: creating tablet
I20250905 19:52:20.169574 1169 ts_tablet_manager.cc:1428] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20250905 19:52:20.097404 1170 tablet_bootstrap.cc:654] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.112087 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "19e4edddafcd4d22a0e0ae66b0b953e0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.112646 1203 raft_consensus.cc:491] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.171710 1203 raft_consensus.cc:513] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.172430 1203 leader_election.cc:290] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.172504 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9093f45e0bc746fba6bea50cc943e765" } modify_external_catalogs: true
I20250905 19:52:20.172698 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9093f45e0bc746fba6bea50cc943e765" } modify_external_catalogs: true
W20250905 19:52:20.173738 848 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.174425 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_885d880838c44eab9ff1220768e59373" } modify_external_catalogs: true
I20250905 19:52:20.174609 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_885d880838c44eab9ff1220768e59373" } modify_external_catalogs: true
I20250905 19:52:20.176440 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.176544 885 tablet_service.cc:1515] Processing DeleteTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.176734 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.177295 1185 tablet_replica.cc:331] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:20.177608 1185 raft_consensus.cc:2241] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.177824 1185 raft_consensus.cc:2270] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.178797 1185 ts_tablet_manager.cc:1905] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:20.189304 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 087172093fff4c41980dc2daeabb07c3 consensus state has no leader
I20250905 19:52:20.192165 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 1f1a8e7059b348cba34069ef49bf1aad
I20250905 19:52:20.193027 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.192978 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_80c2578ee60e4142b0b77d1beba7e122"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.193622 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_80c2578ee60e4142b0b77d1beba7e122 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.195745 1185 ts_tablet_manager.cc:1918] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.196796 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.198218 1169 tablet_bootstrap.cc:492] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:20.199344 711 leader_election.cc:343] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:20.200089 1169 tablet_bootstrap.cc:654] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.195948 1185 log.cc:1199] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:20.205070 1185 ts_tablet_manager.cc:1939] T 6d12c7fbb84849c68274810b3fdcba04 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:20.206395 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6d12c7fbb84849c68274810b3fdcba04 (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:20.206816 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:20.207460 712 leader_election.cc:343] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:20.207692 712 leader_election.cc:304] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:20.208328 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.208961 1598 raft_consensus.cc:2747] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.209343 1169 tablet_bootstrap.cc:492] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.209568 1169 ts_tablet_manager.cc:1397] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.003s
I20250905 19:52:20.210450 1170 tablet_bootstrap.cc:492] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:20.210695 1170 ts_tablet_manager.cc:1397] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.182s user 0.001s sys 0.004s
I20250905 19:52:20.210718 1169 raft_consensus.cc:357] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.210968 1169 raft_consensus.cc:383] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.211083 1169 raft_consensus.cc:738] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.212261 1185 tablet_replica.cc:331] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:20.212550 1185 raft_consensus.cc:2241] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.212627 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.212796 1185 raft_consensus.cc:2270] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.213513 1185 ts_tablet_manager.cc:1905] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:20.214243 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 578a6b9224cc4d8585d6c4b241762e1b already in progress: creating tablet
W20250905 19:52:20.214578 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1a8e7059b348cba34069ef49bf1aad already in progress: creating tablet
I20250905 19:52:20.223086 1186 ts_tablet_manager.cc:1918] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.223263 1186 log.cc:1199] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/59faa6716fed4ea6b563df5f024b2058
I20250905 19:52:20.225494 1169 consensus_queue.cc:260] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.225785 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_d34934e0d3f4442fa41d83040f628e19"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.226366 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d34934e0d3f4442fa41d83040f628e19 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:20.226774 441 consensus_peers.cc:489] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 -> Peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Couldn't send request to peer e38333af29db4bf6af41ee5a25d6701e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250905 19:52:20.229879 1186 ts_tablet_manager.cc:1939] T 59faa6716fed4ea6b563df5f024b2058 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:20.211916 1170 raft_consensus.cc:357] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.232192 1170 raft_consensus.cc:383] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.232298 1170 raft_consensus.cc:738] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.233727 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 59faa6716fed4ea6b563df5f024b2058 (table default.Table_37d80cf1db464cff9b77c0aff7a595c2 [id=f5891226b675435f81cbdca3c8b4f43e]) successfully deleted
I20250905 19:52:20.233940 1170 consensus_queue.cc:260] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.234481 1170 ts_tablet_manager.cc:1428] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250905 19:52:20.236097 1171 tablet_bootstrap.cc:492] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:20.236122 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a5af8232d31240ff855b79dcea29396b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5af8232d31240ff855b79dcea29396b already in progress: creating tablet
I20250905 19:52:20.236276 1171 ts_tablet_manager.cc:1397] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.073s user 0.031s sys 0.026s
W20250905 19:52:20.237133 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1a8e7059b348cba34069ef49bf1aad already in progress: creating tablet
I20250905 19:52:20.237118 1171 raft_consensus.cc:357] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.237313 1171 raft_consensus.cc:383] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.237418 1171 raft_consensus.cc:738] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.237721 1171 consensus_queue.cc:260] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:20.238335 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 578a6b9224cc4d8585d6c4b241762e1b already in progress: creating tablet
I20250905 19:52:20.239857 1169 ts_tablet_manager.cc:1428] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.030s user 0.001s sys 0.002s
I20250905 19:52:20.243117 1171 ts_tablet_manager.cc:1428] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 19:52:20.243605 1169 tablet_bootstrap.cc:492] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.245002 1170 tablet_bootstrap.cc:492] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:20.245985 1171 tablet_bootstrap.cc:492] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:20.249171 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 087172093fff4c41980dc2daeabb07c3 consensus state has no leader
I20250905 19:52:20.249440 1171 tablet_bootstrap.cc:654] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.249778 1169 tablet_bootstrap.cc:654] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.252686 449 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet c8ced81fbfe849888655aca77abe6d74
I20250905 19:52:20.253419 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 44769f1170354870ae2d34fdb67ffbae
I20250905 19:52:20.254467 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 4d2ed1bc09c94526bc9662f2d4d21f8f from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.254797 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.257016 1394 raft_consensus.cc:491] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.257243 1394 raft_consensus.cc:513] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.258059 1394 leader_election.cc:290] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.259418 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.259896 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:20.260252 986 leader_election.cc:343] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.260663 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.260819 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 44769f1170354870ae2d34fdb67ffbae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44769f1170354870ae2d34fdb67ffbae already in progress: creating tablet
W20250905 19:52:20.261380 984 leader_election.cc:343] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.261543 984 leader_election.cc:304] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.262146 1394 raft_consensus.cc:2747] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.269101 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.270509 1169 tablet_bootstrap.cc:492] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.270740 1169 ts_tablet_manager.cc:1397] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250905 19:52:20.271781 1169 raft_consensus.cc:357] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.272073 1169 raft_consensus.cc:383] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.272174 1169 raft_consensus.cc:738] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:20.272917 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 44769f1170354870ae2d34fdb67ffbae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44769f1170354870ae2d34fdb67ffbae already in progress: creating tablet
I20250905 19:52:20.274310 1169 consensus_queue.cc:260] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.278672 1169 ts_tablet_manager.cc:1428] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:20.279177 1169 tablet_bootstrap.cc:492] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.274305 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b12931248d47450dad62ed65df3edf73" } new_table_name: "default.Table_826b7d0620fd47ccb449a28ee9469199" modify_external_catalogs: true
I20250905 19:52:20.274791 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7fa399abb0ac4fbeaeb856e415850864" } new_table_name: "default.Table_ebc7abee432a408886662ed9c0eaa419" modify_external_catalogs: true
I20250905 19:52:20.275560 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_024f98676ba34eaca458c9d9636e3b83"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.282768 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_024f98676ba34eaca458c9d9636e3b83 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.284420 1185 ts_tablet_manager.cc:1918] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.284569 1185 log.cc:1199] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/77cbdda0e2d641bd8092f00925bb9a9c
I20250905 19:52:20.284992 1185 ts_tablet_manager.cc:1939] T 77cbdda0e2d641bd8092f00925bb9a9c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:20.290576 1169 tablet_bootstrap.cc:654] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.293116 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 77cbdda0e2d641bd8092f00925bb9a9c (table default.Table_39151a7d21ac4d49bb5728f518e3db4f [id=e0a1acd20bc340088714693c74f31d2f]) successfully deleted
I20250905 19:52:20.293552 1185 tablet_replica.cc:331] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:20.293949 1185 raft_consensus.cc:2241] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.294193 1185 raft_consensus.cc:2270] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.294763 1185 ts_tablet_manager.cc:1905] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.298003 1600 raft_consensus.cc:491] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.298171 1600 raft_consensus.cc:513] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.298406 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.298919 1600 leader_election.cc:290] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:20.298997 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.299773 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:20.300220 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "66df0522aa2842f68aa824571fa61b05" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.300736 1394 raft_consensus.cc:491] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.300786 712 leader_election.cc:304] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:20.300901 1394 raft_consensus.cc:513] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.301191 1600 raft_consensus.cc:2747] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.301488 1394 leader_election.cc:290] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.302342 1170 tablet_bootstrap.cc:654] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.302454 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.303040 984 leader_election.cc:343] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.303339 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:20.303857 986 leader_election.cc:343] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.304013 986 leader_election.cc:304] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.304723 1394 raft_consensus.cc:2747] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.305032 1584 raft_consensus.cc:491] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.305189 1584 raft_consensus.cc:513] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.305800 1584 leader_election.cc:290] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.306754 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57f851a6eecc4a41806bdaa19cc5afab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.307205 848 leader_election.cc:343] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.307821 849 leader_election.cc:343] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:20.307946 849 leader_election.cc:304] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.308403 1584 raft_consensus.cc:2747] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:20.309544 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89a73e984cbb4268b563f1ca01f4c65d consensus state has no leader
I20250905 19:52:20.309855 1171 tablet_bootstrap.cc:492] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.309991 1169 tablet_bootstrap.cc:492] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.310001 1171 ts_tablet_manager.cc:1397] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.064s user 0.023s sys 0.018s
I20250905 19:52:20.310117 1169 ts_tablet_manager.cc:1397] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250905 19:52:20.310824 1171 raft_consensus.cc:357] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.311005 1171 raft_consensus.cc:383] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.311092 1171 raft_consensus.cc:738] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.311257 1169 raft_consensus.cc:357] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.311421 1169 raft_consensus.cc:383] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.311352 1171 consensus_queue.cc:260] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.311501 1169 raft_consensus.cc:738] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.311955 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_10296fdc41264234b254f3e5ce2be23d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.311775 1169 consensus_queue.cc:260] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:20.313036 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_10296fdc41264234b254f3e5ce2be23d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.315312 1171 ts_tablet_manager.cc:1428] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:20.318603 1169 ts_tablet_manager.cc:1428] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250905 19:52:20.321446 1169 tablet_bootstrap.cc:492] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.323190 1169 tablet_bootstrap.cc:654] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:20.323225 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e58bbb6575fd4437b242032a281914ea consensus state has no leader
I20250905 19:52:20.324566 1171 tablet_bootstrap.cc:492] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.326406 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7a422c26de1b49dfa1ef67add2fc4fe8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.327194 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7a422c26de1b49dfa1ef67add2fc4fe8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.331611 1185 ts_tablet_manager.cc:1918] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.331774 1185 log.cc:1199] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/7edcf57f7753449d8a5fcdb6f4f1dc21
I20250905 19:52:20.335843 1185 ts_tablet_manager.cc:1939] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:20.336819 1171 tablet_bootstrap.cc:654] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.337229 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 (table default.Table_b86b0c6f5fe4496ead402827d2db3f1d [id=8638b095c62741559b2281befee964ea]) successfully deleted
W20250905 19:52:20.338064 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1a8e7059b348cba34069ef49bf1aad already in progress: creating tablet
W20250905 19:52:20.339263 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 44769f1170354870ae2d34fdb67ffbae because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 44769f1170354870ae2d34fdb67ffbae already in progress: creating tablet
I20250905 19:52:20.343482 447 master_service.cc:948] ReplaceTablet: received request to replace tablet baefaa1f042348c1874e306f1e5a8781 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.344923 1169 tablet_bootstrap.cc:492] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.345103 1169 ts_tablet_manager.cc:1397] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.003s
I20250905 19:52:20.346060 1169 raft_consensus.cc:357] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.346257 1169 raft_consensus.cc:383] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.346329 1169 raft_consensus.cc:738] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.346594 1169 consensus_queue.cc:260] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.352231 1169 ts_tablet_manager.cc:1428] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20250905 19:52:20.353408 1169 tablet_bootstrap.cc:492] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.355126 1169 tablet_bootstrap.cc:654] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.355571 1584 raft_consensus.cc:491] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.355747 1584 raft_consensus.cc:513] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.356377 1584 leader_election.cc:290] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:20.357313 849 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:20.358136 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d480531ba94f4b0f9fe40e88fe91512e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.358654 848 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.358765 848 leader_election.cc:304] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.359324 1584 raft_consensus.cc:2747] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.364421 1171 tablet_bootstrap.cc:492] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.364611 1171 ts_tablet_manager.cc:1397] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.003s
I20250905 19:52:20.365370 1171 raft_consensus.cc:357] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.365576 1171 raft_consensus.cc:383] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.365664 1171 raft_consensus.cc:738] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.365908 1171 consensus_queue.cc:260] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.366418 1171 ts_tablet_manager.cc:1428] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:20.369052 1171 tablet_bootstrap.cc:492] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.379582 1169 tablet_bootstrap.cc:492] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.379853 1169 ts_tablet_manager.cc:1397] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.003s
I20250905 19:52:20.381140 1169 raft_consensus.cc:357] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.381393 1169 raft_consensus.cc:383] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.381585 1169 raft_consensus.cc:738] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.381919 1169 consensus_queue.cc:260] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.382194 1170 tablet_bootstrap.cc:492] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:20.382428 1170 ts_tablet_manager.cc:1397] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.137s user 0.004s sys 0.000s
I20250905 19:52:20.383466 1170 raft_consensus.cc:357] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.383724 1170 raft_consensus.cc:383] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.383808 1170 raft_consensus.cc:738] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.383986 1171 tablet_bootstrap.cc:654] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.384166 1170 consensus_queue.cc:260] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.387931 1169 ts_tablet_manager.cc:1428] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 19:52:20.389523 1169 tablet_bootstrap.cc:492] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.391355 1169 tablet_bootstrap.cc:654] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.396796 1171 tablet_bootstrap.cc:492] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.396972 1171 ts_tablet_manager.cc:1397] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.002s
I20250905 19:52:20.397785 1171 raft_consensus.cc:357] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.397982 1171 raft_consensus.cc:383] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.398070 1171 raft_consensus.cc:738] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.398281 1171 consensus_queue.cc:260] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.399161 1169 tablet_bootstrap.cc:492] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.399899 1171 ts_tablet_manager.cc:1428] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:20.400769 1171 tablet_bootstrap.cc:492] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.403398 1171 tablet_bootstrap.cc:654] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.403895 1169 ts_tablet_manager.cc:1397] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250905 19:52:20.403934 1236 raft_consensus.cc:491] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.404383 1394 raft_consensus.cc:491] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.404894 1394 raft_consensus.cc:513] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.405543 1394 leader_election.cc:290] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.405963 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.406337 1494 raft_consensus.cc:491] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.406495 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:20.406777 984 leader_election.cc:343] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.406492 1494 raft_consensus.cc:513] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:20.407106 986 leader_election.cc:343] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.407234 1236 raft_consensus.cc:513] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.407531 1494 leader_election.cc:290] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.408519 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edac849e8834894be2d4ff323edd13d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.409251 986 leader_election.cc:304] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:20.409268 848 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.409790 849 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
I20250905 19:52:20.409914 849 leader_election.cc:304] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.410409 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74a7e7d9874e4d26bda3ab24a6e24adc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.410668 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74a7e7d9874e4d26bda3ab24a6e24adc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.411068 1236 leader_election.cc:290] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:20.411194 986 leader_election.cc:343] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.411391 984 leader_election.cc:343] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.411492 984 leader_election.cc:304] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.411715 1394 raft_consensus.cc:2747] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.412010 1494 raft_consensus.cc:2747] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.412207 1394 raft_consensus.cc:2747] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.412371 1169 raft_consensus.cc:357] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.412549 1169 raft_consensus.cc:383] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.412636 1169 raft_consensus.cc:738] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.412873 1169 consensus_queue.cc:260] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.413347 1169 ts_tablet_manager.cc:1428] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250905 19:52:20.413684 1169 tablet_bootstrap.cc:492] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.417714 1169 tablet_bootstrap.cc:654] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.423599 1170 ts_tablet_manager.cc:1428] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.041s user 0.002s sys 0.001s
I20250905 19:52:20.428710 1494 raft_consensus.cc:491] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.428973 1494 raft_consensus.cc:513] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.429857 1494 leader_election.cc:290] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:20.430939 849 leader_election.cc:343] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:20.431156 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "757a7d3fba8d43dda36474539b7296ba" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.431943 848 leader_election.cc:343] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.432158 848 leader_election.cc:304] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.432773 1494 raft_consensus.cc:2747] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.433573 1171 tablet_bootstrap.cc:492] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.433773 1171 ts_tablet_manager.cc:1397] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20250905 19:52:20.434686 1171 raft_consensus.cc:357] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.434872 1171 raft_consensus.cc:383] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.434957 1171 raft_consensus.cc:738] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.435227 1171 consensus_queue.cc:260] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.436069 1171 ts_tablet_manager.cc:1428] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:20.436389 1171 tablet_bootstrap.cc:492] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.436559 1170 tablet_bootstrap.cc:492] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:20.439293 1171 tablet_bootstrap.cc:654] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.440840 1494 raft_consensus.cc:491] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.440984 1494 raft_consensus.cc:513] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.441553 1494 leader_election.cc:290] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.441807 1170 tablet_bootstrap.cc:654] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.442282 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7c5807c1132747e6af79d93e7496d500" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.443022 848 leader_election.cc:343] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.443293 849 leader_election.cc:343] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 7c5807c1132747e6af79d93e7496d500
I20250905 19:52:20.443444 849 leader_election.cc:304] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.444077 1494 raft_consensus.cc:2747] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.445171 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f93e2f3cbd884fbf84082156519631cb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.446084 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f93e2f3cbd884fbf84082156519631cb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.450412 1169 tablet_bootstrap.cc:492] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.450614 1169 ts_tablet_manager.cc:1397] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.001s
I20250905 19:52:20.450654 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_8e27522da15a4d8ebb55104bcbe5fc5f" } new_table_name: "default.Table_dc9077e09b234b828d9df52479655f50" modify_external_catalogs: true
I20250905 19:52:20.451507 1169 raft_consensus.cc:357] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.451699 1169 raft_consensus.cc:383] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.451783 1169 raft_consensus.cc:738] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.452080 1171 tablet_bootstrap.cc:492] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.452163 1394 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.452224 1171 ts_tablet_manager.cc:1397] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.000s
I20250905 19:52:20.452286 1394 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.453014 1394 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.453088 1171 raft_consensus.cc:357] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.453248 1171 raft_consensus.cc:383] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.453352 1171 raft_consensus.cc:738] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.453611 1171 consensus_queue.cc:260] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.453939 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.454476 1171 ts_tablet_manager.cc:1428] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:20.455246 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.455476 1171 tablet_bootstrap.cc:492] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:20.456002 984 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.456483 986 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.456609 986 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.456960 1236 raft_consensus.cc:491] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.457109 1236 raft_consensus.cc:513] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.457298 1171 tablet_bootstrap.cc:654] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.457744 1236 leader_election.cc:290] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.457906 1236 raft_consensus.cc:2747] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.459827 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0702063ec1c4f6fba58e6d4f3912212" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.462101 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0702063ec1c4f6fba58e6d4f3912212" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:20.462639 984 leader_election.cc:343] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.462949 986 leader_election.cc:343] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.463049 986 leader_election.cc:304] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.463451 1394 raft_consensus.cc:2747] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.465394 1169 consensus_queue.cc:260] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.473104 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:20.474071 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 2f581403ec924fb0948400df41ed8aab (table default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]). Expected version 1 got 0
I20250905 19:52:20.474370 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_f6b3a5016b7045bdbd914ad7ee173c25 [id=fe9667d9d9cf4b0e99651b827e50e4ac]). Expected version 1 got 0
I20250905 19:52:20.474588 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 36f966bf512d4cbe9ae406386db31533 (table default.Table_51169c758cd646cd91f3bda396598e10 [id=d92c116575544c1c8d98befb5a420348]). Expected version 1 got 0
I20250905 19:52:20.474958 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:20.475281 1169 ts_tablet_manager.cc:1428] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
W20250905 19:52:20.475358 453 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:56031 (request call id 57) took 759 ms (client timeout 1000 ms). Trace:
W20250905 19:52:20.475505 453 rpcz_store.cc:269] 0905 19:52:19.715737 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:19.741649 (+ 25912us) service_pool.cc:225] Handling call
0905 19:52:20.475292 (+733643us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":1}
I20250905 19:52:20.476601 1236 raft_consensus.cc:491] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.476626 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.476776 1236 raft_consensus.cc:513] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.476763 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.477242 1171 tablet_bootstrap.cc:492] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.477434 1171 ts_tablet_manager.cc:1397] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.000s
I20250905 19:52:20.477427 1236 leader_election.cc:290] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:20.477429 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.478035 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.478228 1171 raft_consensus.cc:357] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.478456 1171 raft_consensus.cc:383] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.478536 1171 raft_consensus.cc:738] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.478729 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.478859 1171 consensus_queue.cc:260] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.479001 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c20eea1b024d7f93f18f0c089efc42) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:20.479406 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 01c20eea1b024d7f93f18f0c089efc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01c20eea1b024d7f93f18f0c089efc42 already in progress: creating tablet
I20250905 19:52:20.479470 1186 tablet_replica.cc:331] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:20.479676 1171 ts_tablet_manager.cc:1428] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250905 19:52:20.479677 1186 raft_consensus.cc:2241] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.479741 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.479878 1186 raft_consensus.cc:2270] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.480000 1171 tablet_bootstrap.cc:492] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.480129 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 262bcd1cfc994bb58e999b141b9bc7ea
W20250905 19:52:20.480306 984 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.480546 986 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.480701 986 leader_election.cc:304] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.480643 454 catalog_manager.cc:5582] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:20.481110 1236 raft_consensus.cc:2747] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.481119 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]). Expected version 2 got 0
I20250905 19:52:20.481374 454 catalog_manager.cc:5582] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:20.481755 1171 tablet_bootstrap.cc:654] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.481747 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]). Expected version 2 got 0
I20250905 19:52:20.475504 452 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:52586 (request call id 338) took 1360 ms. Trace:
I20250905 19:52:20.482151 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]). Expected version 5 got 0
I20250905 19:52:20.482348 454 catalog_manager.cc:5582] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:20.482676 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_d6e9e58d90324cd2a7110d05c7588160 [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 2 got 0
I20250905 19:52:20.483462 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.483546 1186 ts_tablet_manager.cc:1905] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.483655 1171 tablet_bootstrap.cc:492] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.482213 452 rpcz_store.cc:276] 0905 19:52:19.115250 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:19.400843 (+285593us) service_pool.cc:225] Handling call
0905 19:52:19.431411 (+ 30568us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:19.445400 (+ 13989us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:19.445472 (+ 72us) catalog_manager.cc:2799] Locking tablets
0905 19:52:19.811954 (+366482us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:20.151348 (+339394us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:20.151498 (+ 150us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:20.474882 (+323384us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:20.475459 (+ 577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:20.108700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:20.111365 (+ 2665us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:20.111370 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:20.111373 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:20.111499 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:20.111511 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:20.111518 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:20.111643 (+ 125us) tablet.cc:699] Row locks acquired
0905 19:52:20.111646 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:20.111680 (+ 34us) write_op.cc:270] Start()
0905 19:52:20.111710 (+ 30us) write_op.cc:276] Timestamp: P: 1757101940111661 usec, L: 0
0905 19:52:20.111714 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:20.113107 (+ 1393us) log.cc:844] Serialized 725 byte log entry
0905 19:52:20.150327 (+ 37220us) op_driver.cc:464] REPLICATION: finished
0905 19:52:20.150715 (+ 388us) write_op.cc:301] APPLY: starting
0905 19:52:20.150736 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:20.150783 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:20.150807 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:20.150886 (+ 79us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:20.150933 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:20.150945 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:20.151041 (+ 96us) log.cc:844] Serialized 29 byte log entry
0905 19:52:20.151117 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:20.151138 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:20.151145 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:20.151217 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:20.151221 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":4096,"child_traces":[["op",{"apply.queue_time_us":322,"apply.run_cpu_time_us":503,"apply.run_wall_time_us":502,"num_ops":2,"prepare.queue_time_us":295968,"prepare.run_cpu_time_us":877,"prepare.run_wall_time_us":8103,"raft.queue_time_us":404,"raft.run_cpu_time_us":222,"raft.run_wall_time_us":222,"replication_time_us":38574}]]}
I20250905 19:52:20.484297 1171 ts_tablet_manager.cc:1397] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250905 19:52:20.484419 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_eade9419b7b144aa9c79d6990802cfcb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.484597 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 262bcd1cfc994bb58e999b141b9bc7ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262bcd1cfc994bb58e999b141b9bc7ea already in progress: creating tablet
W20250905 19:52:20.485003 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eade9419b7b144aa9c79d6990802cfcb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.485327 1171 raft_consensus.cc:357] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.485513 1171 raft_consensus.cc:383] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.485590 1171 raft_consensus.cc:738] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.485812 1171 consensus_queue.cc:260] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:20.486238 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 01c20eea1b024d7f93f18f0c089efc42 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 01c20eea1b024d7f93f18f0c089efc42 already in progress: creating tablet
I20250905 19:52:20.486338 1171 ts_tablet_manager.cc:1428] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:20.486598 1171 tablet_bootstrap.cc:492] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.486740 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.488343 1171 tablet_bootstrap.cc:654] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:20.488826 848 consensus_peers.cc:489] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:20.489549 446 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:33943 (request call id 40) took 760 ms (client timeout 1000 ms). Trace:
W20250905 19:52:20.489670 446 rpcz_store.cc:269] 0905 19:52:19.729368 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:19.743465 (+ 14097us) service_pool.cc:225] Handling call
0905 19:52:20.489502 (+746037us) inbound_call.cc:173] Queueing success response
Metrics: {"reported_tablets":11}
I20250905 19:52:20.490288 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.490563 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.490798 1184 tablet_replica.cc:331] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:20.490885 1169 tablet_bootstrap.cc:492] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.491917 1186 ts_tablet_manager.cc:1918] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.492272 446 master_service.cc:948] ReplaceTablet: received request to replace tablet a5f076d9917f40d899d7422ad974fe0f from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.492759 1184 raft_consensus.cc:2241] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.492975 1184 raft_consensus.cc:2270] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.493750 1186 log.cc:1199] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:20.494306 1186 ts_tablet_manager.cc:1939] T 57259d8eb0d44d1cbac80047c7ad67b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:20.495102 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 57259d8eb0d44d1cbac80047c7ad67b6 (table default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]) successfully deleted
W20250905 19:52:20.496057 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 262bcd1cfc994bb58e999b141b9bc7ea because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 262bcd1cfc994bb58e999b141b9bc7ea already in progress: creating tablet
I20250905 19:52:20.496775 744 tablet_service.cc:1468] Processing CreateTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 (DEFAULT_TABLE table=default.Table_88ff02c5dce2458784f09b64cee688f1 [id=378b42fcb4e741e4b4755deb4b0ef51a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.497277 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de936d2309a411eabbd59f7b73ce7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.499060 889 tablet_service.cc:1468] Processing CreateTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 (DEFAULT_TABLE table=default.Table_88ff02c5dce2458784f09b64cee688f1 [id=378b42fcb4e741e4b4755deb4b0ef51a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.499830 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de936d2309a411eabbd59f7b73ce7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.502903 753 tablet_service.cc:1468] Processing CreateTablet for tablet dc95a83ca872495894ef439cdda4e364 (DEFAULT_TABLE table=default.Table_79a0ddd169324b64b69f90b3d65bfa11 [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.503386 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc95a83ca872495894ef439cdda4e364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.503391 1184 ts_tablet_manager.cc:1905] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.503417 886 tablet_service.cc:1468] Processing CreateTablet for tablet ea6df8634a314ceabcf3fd004bf7e241 (DEFAULT_TABLE table=default.Table_a8a69461d6db47e298856ec5bc896165 [id=ea980bda9f2f41c8bf82af3ce10d7151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.504030 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6df8634a314ceabcf3fd004bf7e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.508338 746 tablet_service.cc:1468] Processing CreateTablet for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 (DEFAULT_TABLE table=default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.508507 885 tablet_service.cc:1468] Processing CreateTablet for tablet dc95a83ca872495894ef439cdda4e364 (DEFAULT_TABLE table=default.Table_79a0ddd169324b64b69f90b3d65bfa11 [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.508782 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.509068 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc95a83ca872495894ef439cdda4e364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.512416 739 tablet_service.cc:1468] Processing CreateTablet for tablet b3ea2e2059a4425892fa3c0757e69e4c (DEFAULT_TABLE table=default.Table_e9cdd0c9bd8840658a9c9aeb2385265a [id=3e181278985a4082a8173307e0a028d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.512684 755 tablet_service.cc:1468] Processing CreateTablet for tablet 0fb8fe93061848a0b055979bad10f08b (DEFAULT_TABLE table=default.Table_fe588d1872d847a7b9cd61b07c6f7abb [id=4925822c977c4a4588c9fe6117ee722e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.513005 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ea2e2059a4425892fa3c0757e69e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.517275 743 tablet_service.cc:1468] Processing CreateTablet for tablet ea6df8634a314ceabcf3fd004bf7e241 (DEFAULT_TABLE table=default.Table_a8a69461d6db47e298856ec5bc896165 [id=ea980bda9f2f41c8bf82af3ce10d7151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.518816 738 tablet_service.cc:1468] Processing CreateTablet for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3 (DEFAULT_TABLE table=default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.519639 754 tablet_service.cc:1468] Processing CreateTablet for tablet dcf448f9abc348df8d2d2501420d8f45 (DEFAULT_TABLE table=default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.520886 741 tablet_service.cc:1468] Processing CreateTablet for tablet 4dd2a96b8180438fbc722be1c18a0189 (DEFAULT_TABLE table=default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.502941 750 tablet_service.cc:1468] Processing CreateTablet for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (DEFAULT_TABLE table=default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.521438 740 tablet_service.cc:1468] Processing CreateTablet for tablet 5bd9380e8ffc432da4a684934c2df68f (DEFAULT_TABLE table=default.Table_a21b18f7481c4713823fadf4b164b7bf [id=16bc518342ba416b9ef21427ac1429f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.522223 747 tablet_service.cc:1468] Processing CreateTablet for tablet 31e332cdfedd4ff6b5026baa73e2caf3 (DEFAULT_TABLE table=default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.522545 879 tablet_service.cc:1468] Processing CreateTablet for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 (DEFAULT_TABLE table=default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.523003 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.523983 888 tablet_service.cc:1468] Processing CreateTablet for tablet b3ea2e2059a4425892fa3c0757e69e4c (DEFAULT_TABLE table=default.Table_e9cdd0c9bd8840658a9c9aeb2385265a [id=3e181278985a4082a8173307e0a028d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.524529 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ea2e2059a4425892fa3c0757e69e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.528384 893 tablet_service.cc:1468] Processing CreateTablet for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (DEFAULT_TABLE table=default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.528843 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3315d3e9633e42ea8d7a6b5934cb0438. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.532294 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6df8634a314ceabcf3fd004bf7e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.532449 878 tablet_service.cc:1468] Processing CreateTablet for tablet 0fb8fe93061848a0b055979bad10f08b (DEFAULT_TABLE table=default.Table_fe588d1872d847a7b9cd61b07c6f7abb [id=4925822c977c4a4588c9fe6117ee722e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.532951 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb8fe93061848a0b055979bad10f08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.536142 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e332cdfedd4ff6b5026baa73e2caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.536621 883 tablet_service.cc:1468] Processing CreateTablet for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3 (DEFAULT_TABLE table=default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.537062 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.540449 880 tablet_service.cc:1468] Processing CreateTablet for tablet dcf448f9abc348df8d2d2501420d8f45 (DEFAULT_TABLE table=default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.541052 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf448f9abc348df8d2d2501420d8f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.544390 890 tablet_service.cc:1468] Processing CreateTablet for tablet 4dd2a96b8180438fbc722be1c18a0189 (DEFAULT_TABLE table=default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.544732 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd2a96b8180438fbc722be1c18a0189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.544973 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd2a96b8180438fbc722be1c18a0189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.548431 887 tablet_service.cc:1468] Processing CreateTablet for tablet 5bd9380e8ffc432da4a684934c2df68f (DEFAULT_TABLE table=default.Table_a21b18f7481c4713823fadf4b164b7bf [id=16bc518342ba416b9ef21427ac1429f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.548990 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd9380e8ffc432da4a684934c2df68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.552428 891 tablet_service.cc:1468] Processing CreateTablet for tablet 31e332cdfedd4ff6b5026baa73e2caf3 (DEFAULT_TABLE table=default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.552959 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e332cdfedd4ff6b5026baa73e2caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.557614 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd9380e8ffc432da4a684934c2df68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.560171 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 8ed6109bdf8f4caf8c223a42401b94ab from {username='slave'} at 127.0.0.1:52586
W20250905 19:52:20.560735 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet efceaf3c7bda44ddabd5d38130af3dd2 consensus state has no leader
I20250905 19:52:20.561486 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:20.562027 447 catalog_manager.cc:6652] ReplaceTablet: tablet baefaa1f042348c1874e306f1e5a8781 deleted and replaced by tablet 1e9b7458f96148b08962a9b82223187e
I20250905 19:52:20.562319 447 master_service.cc:948] ReplaceTablet: received request to replace tablet ac5c88cf8d15433a940e937f0a225582 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.565826 882 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:20.567036 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet baefaa1f042348c1874e306f1e5a8781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet baefaa1f042348c1874e306f1e5a8781 already in progress: creating tablet
I20250905 19:52:20.576229 1586 raft_consensus.cc:491] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.576424 1586 raft_consensus.cc:513] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.577169 1586 leader_election.cc:290] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.589824 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb8fe93061848a0b055979bad10f08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.590240 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f581403ec924fb0948400df41ed8aab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.591102 848 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.592439 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf448f9abc348df8d2d2501420d8f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.595132 756 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.601657 744 tablet_service.cc:1468] Processing CreateTablet for tablet 2350bd08f977424ba94ee8fa01f66efa (DEFAULT_TABLE table=default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.603441 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3315d3e9633e42ea8d7a6b5934cb0438. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:20.604656 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet baefaa1f042348c1874e306f1e5a8781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet baefaa1f042348c1874e306f1e5a8781 already in progress: creating tablet
W20250905 19:52:20.605118 848 consensus_peers.cc:489] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:20.606227 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2350bd08f977424ba94ee8fa01f66efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.610473 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.619776 1586 consensus_queue.cc:1035] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:20.622267 1586 raft_consensus.cc:491] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.622465 1586 raft_consensus.cc:513] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.623333 1586 leader_election.cc:290] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.624743 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af5748bea3bd4f51a0d6c28a1d9044a0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.625411 848 leader_election.cc:343] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.626257 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.629060 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.630069 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.630317 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af5748bea3bd4f51a0d6c28a1d9044a0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:20.631956 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 (DEFAULT_TABLE table=default.Table_88ff02c5dce2458784f09b64cee688f1 [id=378b42fcb4e741e4b4755deb4b0ef51a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.629674 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f581403ec924fb0948400df41ed8aab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:20.633502 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3de936d2309a411eabbd59f7b73ce7f3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.637095 1236 raft_consensus.cc:491] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.637408 1052 raft_consensus.cc:2466] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:20.638674 849 leader_election.cc:304] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.639734 1586 raft_consensus.cc:2802] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:20.639873 1586 raft_consensus.cc:491] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:20.639997 1586 raft_consensus.cc:3058] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.645586 1053 raft_consensus.cc:2466] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:20.647729 849 leader_election.cc:304] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.648610 1584 raft_consensus.cc:2802] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:20.648770 1584 raft_consensus.cc:491] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:20.648901 1584 raft_consensus.cc:3058] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.653923 1584 raft_consensus.cc:513] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.654320 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ea6df8634a314ceabcf3fd004bf7e241 (DEFAULT_TABLE table=default.Table_a8a69461d6db47e298856ec5bc896165 [id=ea980bda9f2f41c8bf82af3ce10d7151]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.655620 1028 tablet_service.cc:1468] Processing CreateTablet for tablet dc95a83ca872495894ef439cdda4e364 (DEFAULT_TABLE table=default.Table_79a0ddd169324b64b69f90b3d65bfa11 [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.656389 1236 raft_consensus.cc:513] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.657334 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b3ea2e2059a4425892fa3c0757e69e4c (DEFAULT_TABLE table=default.Table_e9cdd0c9bd8840658a9c9aeb2385265a [id=3e181278985a4082a8173307e0a028d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.657891 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 (DEFAULT_TABLE table=default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.658432 1236 leader_election.cc:290] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.659320 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe94d1254d04949aa25346d8e323c3b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.660092 984 leader_election.cc:343] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.663352 744 tablet_service.cc:1468] Processing CreateTablet for tablet 239539ac19124de192cadae494720a2b (DEFAULT_TABLE table=default.Table_6a7e530e3936440fa898546212620a3b [id=cf56ce3a388047dc9bd2d846817fef73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.663771 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 239539ac19124de192cadae494720a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.664170 1032 tablet_service.cc:1468] Processing CreateTablet for tablet dcf448f9abc348df8d2d2501420d8f45 (DEFAULT_TABLE table=default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.664139 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3 (DEFAULT_TABLE table=default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.665705 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea6df8634a314ceabcf3fd004bf7e241. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.666328 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (DEFAULT_TABLE table=default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.668115 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.667225 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 0fb8fe93061848a0b055979bad10f08b (DEFAULT_TABLE table=default.Table_fe588d1872d847a7b9cd61b07c6f7abb [id=4925822c977c4a4588c9fe6117ee722e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.669941 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dcf448f9abc348df8d2d2501420d8f45. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.674497 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b3ea2e2059a4425892fa3c0757e69e4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.675647 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 31e332cdfedd4ff6b5026baa73e2caf3 (DEFAULT_TABLE table=default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.676096 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 4dd2a96b8180438fbc722be1c18a0189 (DEFAULT_TABLE table=default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.678896 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 5bd9380e8ffc432da4a684934c2df68f (DEFAULT_TABLE table=default.Table_a21b18f7481c4713823fadf4b164b7bf [id=16bc518342ba416b9ef21427ac1429f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.683501 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.689002 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.690294 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5bd9380e8ffc432da4a684934c2df68f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.691586 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31e332cdfedd4ff6b5026baa73e2caf3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.696758 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 239539ac19124de192cadae494720a2b (DEFAULT_TABLE table=default.Table_6a7e530e3936440fa898546212620a3b [id=cf56ce3a388047dc9bd2d846817fef73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.698118 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4dd2a96b8180438fbc722be1c18a0189. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.698700 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 2350bd08f977424ba94ee8fa01f66efa (DEFAULT_TABLE table=default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.699625 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a3ae86cb11e24d348e2f12e982f3ab9f with delete_type TABLET_DATA_DELETED (replaced by tablet 544dffc8b77049c295921d0af091b17d) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.699879 1186 tablet_replica.cc:331] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:20.700112 1186 raft_consensus.cc:2241] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.700328 1186 raft_consensus.cc:2270] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.700284 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_44155b42e0d04fc0b3a53308c495f845" } modify_external_catalogs: true
I20250905 19:52:20.700486 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_44155b42e0d04fc0b3a53308c495f845" } modify_external_catalogs: true
I20250905 19:52:20.700976 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0fb8fe93061848a0b055979bad10f08b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.701422 1186 ts_tablet_manager.cc:1905] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.701941 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_760e5114bbcd4203a4de6891f9510b66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.702543 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_760e5114bbcd4203a4de6891f9510b66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.703735 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_460fbdca606041adbd01239f6cfff523" } modify_external_catalogs: true
I20250905 19:52:20.703876 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_460fbdca606041adbd01239f6cfff523" } modify_external_catalogs: true
I20250905 19:52:20.704823 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2350bd08f977424ba94ee8fa01f66efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.705888 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_36a3196459524d63aafe6ab49f9faf56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.706115 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3315d3e9633e42ea8d7a6b5934cb0438. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:20.706475 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_36a3196459524d63aafe6ab49f9faf56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.710851 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_dccb2b41c5664fcda055f919170fe6cd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.711407 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_dccb2b41c5664fcda055f919170fe6cd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.712018 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e9cdd0c9bd8840658a9c9aeb2385265a" } new_table_name: "default.Table_6e52652f409647a88e809923474b09dd" modify_external_catalogs: true
I20250905 19:52:20.712518 1236 raft_consensus.cc:491] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.712697 1236 raft_consensus.cc:513] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.713554 1236 leader_election.cc:290] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.714322 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ff5415e10957422498084d7730365989" } modify_external_catalogs: true
I20250905 19:52:20.714351 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262bcd1cfc994bb58e999b141b9bc7ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.714547 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ff5415e10957422498084d7730365989" } modify_external_catalogs: true
W20250905 19:52:20.715018 984 leader_election.cc:343] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.716317 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 57259d8eb0d44d1cbac80047c7ad67b6 with delete_type TABLET_DATA_DELETED (replaced by tablet 01c20eea1b024d7f93f18f0c089efc42) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.716828 1236 raft_consensus.cc:491] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.717020 1236 raft_consensus.cc:513] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.717799 1236 leader_election.cc:290] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.718562 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.719157 984 leader_election.cc:343] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.719403 1186 ts_tablet_manager.cc:1918] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.719563 1186 log.cc:1199] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a3ae86cb11e24d348e2f12e982f3ab9f
I20250905 19:52:20.719946 1186 ts_tablet_manager.cc:1939] T a3ae86cb11e24d348e2f12e982f3ab9f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:20.723131 1169 tablet_bootstrap.cc:654] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.724419 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a3ae86cb11e24d348e2f12e982f3ab9f (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]) successfully deleted
W20250905 19:52:20.724767 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 57259d8eb0d44d1cbac80047c7ad67b6 because the tablet was not found. No further retry: Not found: Tablet not found: 57259d8eb0d44d1cbac80047c7ad67b6
I20250905 19:52:20.734442 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:20.735028 446 catalog_manager.cc:6652] ReplaceTablet: tablet a5f076d9917f40d899d7422ad974fe0f deleted and replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:20.735395 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_deab3e4fa1104b67b031b82f945d2917" } new_table_name: "default.Table_e93ef86856c54ccdae803a15856cbc68" modify_external_catalogs: true
I20250905 19:52:20.736418 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ffa14750f1b94c589779f7da0e764ccc
I20250905 19:52:20.736811 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 723168b075874dfcb3eb50c5134c499e
I20250905 19:52:20.737178 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_30534826add54f68973738b8e5c471f0" } new_table_name: "default.Table_c815e30da2544766834b52eaefab63c4" modify_external_catalogs: true
I20250905 19:52:20.737423 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_d66d859b633f453393a574fa8885329b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.737893 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d66d859b633f453393a574fa8885329b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:20.738361 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet b3ea2e2059a4425892fa3c0757e69e4c consensus state has no leader
I20250905 19:52:20.738724 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9c9640e097254520bd3c753e09ca7733"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.739151 451 master_service.cc:948] ReplaceTablet: received request to replace tablet 36f966bf512d4cbe9ae406386db31533 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.739496 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:20.739840 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f96871d807144eeebd6dd5559a55303e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.740175 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a414c05c2c0c40b8a55a7776f36ecdb7" } modify_external_catalogs: true
W20250905 19:52:20.740286 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f96871d807144eeebd6dd5559a55303e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.740319 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a414c05c2c0c40b8a55a7776f36ecdb7" } modify_external_catalogs: true
I20250905 19:52:20.663910 879 tablet_service.cc:1468] Processing CreateTablet for tablet 2350bd08f977424ba94ee8fa01f66efa (DEFAULT_TABLE table=default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.741494 1584 leader_election.cc:290] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.741736 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2350bd08f977424ba94ee8fa01f66efa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.742334 1170 tablet_bootstrap.cc:492] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:20.742480 1170 ts_tablet_manager.cc:1397] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.307s user 0.005s sys 0.001s
I20250905 19:52:20.743255 1170 raft_consensus.cc:357] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.743417 1170 raft_consensus.cc:383] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.743511 1170 raft_consensus.cc:738] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.743798 1170 consensus_queue.cc:260] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.744321 1170 ts_tablet_manager.cc:1428] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:20.745134 1170 tablet_bootstrap.cc:492] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:20.669106 882 tablet_service.cc:1468] Processing CreateTablet for tablet 239539ac19124de192cadae494720a2b (DEFAULT_TABLE table=default.Table_6a7e530e3936440fa898546212620a3b [id=cf56ce3a388047dc9bd2d846817fef73]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.746465 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 239539ac19124de192cadae494720a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.746740 1170 tablet_bootstrap.cc:654] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.748615 744 tablet_service.cc:1515] Processing DeleteTablet for tablet ffa14750f1b94c589779f7da0e764ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.749146 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.749609 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.742538 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f581403ec924fb0948400df41ed8aab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:20.743479 1018 tablet_service.cc:1515] Processing DeleteTablet for tablet ffa14750f1b94c589779f7da0e764ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.743553 1017 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.752250 1052 raft_consensus.cc:3058] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.743956 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.693140 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.693598 894 tablet_service.cc:1515] Processing DeleteTablet for tablet e6f837fb7293474b8ad357f52e42e42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.694278 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c5807c1132747e6af79d93e7496d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.694676 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.695183 877 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.755609 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:20.756215 447 catalog_manager.cc:6652] ReplaceTablet: tablet ac5c88cf8d15433a940e937f0a225582 deleted and replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7
I20250905 19:52:20.756587 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b42374e796034b1cbfc8659587477d3c" } modify_external_catalogs: true
I20250905 19:52:20.756717 1052 raft_consensus.cc:2466] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:20.756786 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b42374e796034b1cbfc8659587477d3c" } modify_external_catalogs: true
I20250905 19:52:20.757542 849 leader_election.cc:304] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:20.757956 1236 raft_consensus.cc:491] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.752220 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.695206 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab with delete_type TABLET_DATA_DELETED (replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.758919 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.759161 892 tablet_service.cc:1515] Processing DeleteTablet for tablet ffa14750f1b94c589779f7da0e764ccc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.759199 894 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.759583 877 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.758096 1236 raft_consensus.cc:513] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.711514 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe94d1254d04949aa25346d8e323c3b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.716076 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262bcd1cfc994bb58e999b141b9bc7ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.734258 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.760658 1185 tablet_replica.cc:331] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:20.760648 1236 leader_election.cc:290] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.760874 744 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.760828 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.761349 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.761787 1186 tablet_replica.cc:331] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:20.761934 1654 raft_consensus.cc:491] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:20.761950 984 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.761978 1186 raft_consensus.cc:2241] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:20.762066 1654 raft_consensus.cc:513] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:20.739164 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c9640e097254520bd3c753e09ca7733 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.762321 1186 raft_consensus.cc:2270] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.762697 1654 leader_election.cc:290] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.762867 1654 raft_consensus.cc:2802] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:20.762934 1186 ts_tablet_manager.cc:1905] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.762987 1654 raft_consensus.cc:695] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:20.763244 1654 consensus_queue.cc:237] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.763548 738 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.764956 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2f581403ec924fb0948400df41ed8aab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:20.765054 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9477afb5b47c46d4b24128a6c4bdabc6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.766870 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.766916 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:20.767279 986 leader_election.cc:343] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.767470 986 leader_election.cc:304] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:20.768606 986 leader_election.cc:343] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.769177 986 leader_election.cc:304] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.770136 1236 raft_consensus.cc:2747] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.770640 1236 raft_consensus.cc:2747] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.643774 1586 raft_consensus.cc:513] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.773510 1586 leader_election.cc:290] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:20.774083 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61667a379974487195f81043683b9a24 consensus state has no leader
W20250905 19:52:20.775485 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 54628965b82e470d811eff95ffc56c5f consensus state has no leader
I20250905 19:52:20.784719 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_890379e1838248739aa892ccc074b89a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.785431 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_890379e1838248739aa892ccc074b89a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.786068 1586 consensus_queue.cc:1035] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:20.786547 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ebc7abee432a408886662ed9c0eaa419" } new_table_name: "default.Table_a9dc5d6bf42c4c8f852069c46921e828" modify_external_catalogs: true
I20250905 19:52:20.788529 738 tablet_service.cc:1515] Processing DeleteTablet for tablet a884eca0e54a44dc920114e61a199689 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.789240 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_d136ca0885e44597a9298e4b7679fdf6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.789778 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d136ca0885e44597a9298e4b7679fdf6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.791642 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:20.792430 738 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:20.798703 986 leader_election.cc:343] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.798861 986 leader_election.cc:304] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:20.799252 986 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.799351 986 leader_election.cc:304] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.803194 454 master_service.cc:948] ReplaceTablet: received request to replace tablet b13d282e1dc54d1faa2882c9b39d17cb from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.804855 1586 raft_consensus.cc:491] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.805097 1586 raft_consensus.cc:513] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.806638 1586 leader_election.cc:290] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.809772 885 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.810216 1186 ts_tablet_manager.cc:1918] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:20.810364 1186 log.cc:1199] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ffa14750f1b94c589779f7da0e764ccc
I20250905 19:52:20.810904 1186 ts_tablet_manager.cc:1939] T ffa14750f1b94c589779f7da0e764ccc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:20.811754 1586 raft_consensus.cc:491] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.812028 1586 raft_consensus.cc:513] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.812918 1586 leader_election.cc:290] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:20.814605 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af5748bea3bd4f51a0d6c28a1d9044a0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:20.815586 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3602094b96c14e66b997e8035fc53c40" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.816218 1657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:34395 every 8 connection(s)
I20250905 19:52:20.816381 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "43111a3512fb4fc0bdf4ee245b16695b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.817693 848 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.819010 848 leader_election.cc:343] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.819940 848 leader_election.cc:343] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.820235 1262 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:34395
W20250905 19:52:20.820808 848 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.822790 1262 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
I20250905 19:52:20.824435 1394 raft_consensus.cc:2747] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.824620 1394 raft_consensus.cc:2747] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.825729 1236 raft_consensus.cc:491] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.826087 1236 raft_consensus.cc:513] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.827097 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6109bdf8f4caf8c223a42401b94ab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:20.827644 984 leader_election.cc:343] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.828016 1236 leader_election.cc:290] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.829128 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 1262
I20250905 19:52:20.830953 1169 tablet_bootstrap.cc:492] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:20.831146 1169 ts_tablet_manager.cc:1397] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.340s user 0.001s sys 0.005s
W20250905 19:52:20.832146 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e58bbb6575fd4437b242032a281914ea consensus state has no leader
I20250905 19:52:20.832089 1169 raft_consensus.cc:357] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.832334 1169 raft_consensus.cc:383] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.832414 1169 raft_consensus.cc:738] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.844887 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.854679 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ffa14750f1b94c589779f7da0e764ccc (table default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685]) successfully deleted
W20250905 19:52:20.855697 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 723168b075874dfcb3eb50c5134c499e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 723168b075874dfcb3eb50c5134c499e already in progress: creating tablet
W20250905 19:52:20.856868 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 19e4edddafcd4d22a0e0ae66b0b953e0 already in progress: creating tablet
W20250905 19:52:20.857898 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ac5c88cf8d15433a940e937f0a225582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac5c88cf8d15433a940e937f0a225582 already in progress: creating tablet
W20250905 19:52:20.858827 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a5f076d9917f40d899d7422ad974fe0f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a5f076d9917f40d899d7422ad974fe0f already in progress: creating tablet
I20250905 19:52:20.861490 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc95a83ca872495894ef439cdda4e364. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.764361 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9477afb5b47c46d4b24128a6c4bdabc6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:20.862228 1050 raft_consensus.cc:2466] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:20.775449 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af5748bea3bd4f51a0d6c28a1d9044a0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:20.863811 1049 raft_consensus.cc:3058] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.864264 1186 tablet_replica.cc:331] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:20.864521 1186 raft_consensus.cc:2241] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.865260 1186 raft_consensus.cc:2270] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.866602 1186 ts_tablet_manager.cc:1905] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.832739 1169 consensus_queue.cc:260] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.875834 1169 ts_tablet_manager.cc:1428] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.045s user 0.003s sys 0.000s
I20250905 19:52:20.876271 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 239539ac19124de192cadae494720a2b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.879087 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ee3eaf9eb84d435187c73a394308861b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.879753 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ee3eaf9eb84d435187c73a394308861b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:20.821641 848 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:20.881208 849 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:20.881458 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_eade9419b7b144aa9c79d6990802cfcb" } new_table_name: "default.Table_f88ff074d1014b9a8a85de3e399792f6" modify_external_catalogs: true
I20250905 19:52:20.881636 849 leader_election.cc:304] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.845774 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6109bdf8f4caf8c223a42401b94ab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:20.881903 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 53513.9us
I20250905 19:52:20.850286 1171 tablet_bootstrap.cc:492] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:20.882344 1171 ts_tablet_manager.cc:1397] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.396s user 0.063s sys 0.145s
W20250905 19:52:20.882304 849 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:20.882503 849 leader_election.cc:304] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.882896 849 leader_election.cc:304] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.883205 1171 raft_consensus.cc:357] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.883452 1171 raft_consensus.cc:383] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.883574 1171 raft_consensus.cc:738] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.883864 1654 raft_consensus.cc:2747] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.884246 1654 raft_consensus.cc:2747] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.884472 1654 raft_consensus.cc:2802] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:20.884744 1654 raft_consensus.cc:491] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:20.883872 1171 consensus_queue.cc:260] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.885144 1171 ts_tablet_manager.cc:1428] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:20.885491 1171 tablet_bootstrap.cc:492] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:20.887921 1171 tablet_bootstrap.cc:654] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.889230 1654 raft_consensus.cc:3058] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:20.894744 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d011d1ec8f634386872e7ab257de3187 has no consensus state
I20250905 19:52:20.897305 1169 tablet_bootstrap.cc:492] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:20.900266 1049 raft_consensus.cc:2466] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:20.909404 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 1 got 0
I20250905 19:52:20.909739 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 2f581403ec924fb0948400df41ed8aab (table default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]). Expected version 1 got 0
I20250905 19:52:20.909758 1236 raft_consensus.cc:491] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.910082 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]). Expected version 1 got 0
I20250905 19:52:20.910221 450 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:20.912879 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.913616 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.913874 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:20.914229 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 92716.5us
I20250905 19:52:20.914281 889 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.919471 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 36f966bf512d4cbe9ae406386db31533
I20250905 19:52:20.920001 451 catalog_manager.cc:6652] ReplaceTablet: tablet 36f966bf512d4cbe9ae406386db31533 deleted and replaced by tablet 3fe2b75a3e59437c885abc18614f1da2
I20250905 19:52:20.920301 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 36f966bf512d4cbe9ae406386db31533 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fe2b75a3e59437c885abc18614f1da2) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.921209 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 36f966bf512d4cbe9ae406386db31533 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fe2b75a3e59437c885abc18614f1da2) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.923332 452 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 29c2b030fafd44ad9e648b2527a4ad07 (table default.Table_f43b3348e3084812974970a576a6fd15 [id=bbdef6ec6956464f9cdd00b3fb506c97]). Expected version 1 got 0
I20250905 19:52:20.923694 452 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:20.924046 452 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0267ac3e3f1a45978807737eca5d91ef (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]). Expected version 1 got 0
I20250905 19:52:20.924345 452 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_05541d16561646a2b97139e3503cf120 [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 1 got 0
I20250905 19:52:20.924623 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 2c884954e5dc4445a12252953593d0bc
I20250905 19:52:20.927554 1236 raft_consensus.cc:513] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.927893 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c884954e5dc4445a12252953593d0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:20.928592 452 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 0267ac3e3f1a45978807737eca5d91ef (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0267ac3e3f1a45978807737eca5d91ef consensus state has no leader
I20250905 19:52:20.929718 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c884954e5dc4445a12252953593d0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.931241 738 tablet_service.cc:1468] Processing CreateTablet for tablet 8092952d755848ccaba3a2ad94b66dad (DEFAULT_TABLE table=default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.931718 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8092952d755848ccaba3a2ad94b66dad. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:20.932610 986 leader_election.cc:343] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.934302 447 master_service.cc:948] ReplaceTablet: received request to replace tablet 4c76077e4adf45a7adc736b0e69032eb from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:20.935636 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_d76eee5aa7234b76a0f3ac551230b3ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:20.936424 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d76eee5aa7234b76a0f3ac551230b3ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.938779 1185 raft_consensus.cc:2241] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.938990 1185 raft_consensus.cc:2270] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.939558 1185 ts_tablet_manager.cc:1905] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.948032 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:20.948566 454 catalog_manager.cc:6652] ReplaceTablet: tablet b13d282e1dc54d1faa2882c9b39d17cb deleted and replaced by tablet c14dc571b586405f8fd1eaa452998978
I20250905 19:52:20.949824 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5f076d9917f40d899d7422ad974fe0f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.950457 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.950598 738 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:20.951156 1236 leader_election.cc:290] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:20.952306 1185 ts_tablet_manager.cc:1918] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.952427 1185 log.cc:1199] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:20.952813 1185 ts_tablet_manager.cc:1939] T 757a7d3fba8d43dda36474539b7296ba P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:20.953434 984 leader_election.cc:343] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.953564 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_48c2edbf02444d7580fff76f96c9c655" } modify_external_catalogs: true
I20250905 19:52:20.953518 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_29345948e72a432f9d41b63ca0c45110"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.953752 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_48c2edbf02444d7580fff76f96c9c655" } modify_external_catalogs: true
W20250905 19:52:20.954188 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_29345948e72a432f9d41b63ca0c45110 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.955320 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2af22804e18e430ea738862d0bade226" } modify_external_catalogs: true
I20250905 19:52:20.955273 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4e5dec3aaf854f29aeda797c8cd3d811"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:20.955466 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2af22804e18e430ea738862d0bade226" } modify_external_catalogs: true
W20250905 19:52:20.955849 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e5dec3aaf854f29aeda797c8cd3d811 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:20.957086 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_89a831c4cad3457b86b01609bfa3a57b" } modify_external_catalogs: true
I20250905 19:52:20.957049 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3f876cdc8a6e4380a81a2b4d49cfb138" } new_table_name: "default.Table_b4a0d3409e7648df827261b499b9f127" modify_external_catalogs: true
I20250905 19:52:20.957221 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_89a831c4cad3457b86b01609bfa3a57b" } modify_external_catalogs: true
I20250905 19:52:20.958547 1654 raft_consensus.cc:513] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.959125 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 757a7d3fba8d43dda36474539b7296ba (table default.Table_a5a47a603b674a41b47e7bf4b55b6a1b [id=106f635aee6b4cd4a7f7b1af9088a286]) successfully deleted
I20250905 19:52:20.959491 1185 tablet_replica.cc:331] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:20.959797 1185 raft_consensus.cc:2241] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:20.960067 1185 raft_consensus.cc:2270] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:20.960703 1185 ts_tablet_manager.cc:1905] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:20.953830 986 leader_election.cc:304] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:20.961673 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 63523.5us
I20250905 19:52:20.961854 1236 raft_consensus.cc:2747] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.962879 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5f076d9917f40d899d7422ad974fe0f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.962953 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.963327 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.963459 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.963673 1236 raft_consensus.cc:491] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:20.963822 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.963851 1236 raft_consensus.cc:513] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.964197 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.964551 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.964598 1236 leader_election.cc:290] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:20.964928 986 leader_election.cc:343] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.965085 986 leader_election.cc:304] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.965428 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29c2b030fafd44ad9e648b2527a4ad07" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:20.965562 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 2c884954e5dc4445a12252953593d0bc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:20.965970 984 leader_election.cc:343] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.966250 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "29c2b030fafd44ad9e648b2527a4ad07" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:20.966784 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.967037 889 tablet_service.cc:1468] Processing CreateTablet for tablet 8092952d755848ccaba3a2ad94b66dad (DEFAULT_TABLE table=default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.967140 1236 raft_consensus.cc:2747] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.967231 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 36f966bf512d4cbe9ae406386db31533 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fe2b75a3e59437c885abc18614f1da2) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.967455 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8092952d755848ccaba3a2ad94b66dad. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:20.967439 986 leader_election.cc:343] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:20.967563 986 leader_election.cc:304] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.967996 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:20.974375 1675 tablet.cc:1722] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:20.978778 1098 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:20.979118 1170 tablet_bootstrap.cc:492] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:20.979039 738 tablet_service.cc:1468] Processing CreateTablet for tablet 4d6f2e44855d4312b1c0367145e3cee8 (DEFAULT_TABLE table=default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.979277 1170 ts_tablet_manager.cc:1397] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.235s user 0.004s sys 0.007s
I20250905 19:52:20.979701 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6f2e44855d4312b1c0367145e3cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.981582 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:20.980346 1170 raft_consensus.cc:357] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.983055 1170 raft_consensus.cc:383] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:20.983057 849 leader_election.cc:304] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:20.983193 1170 raft_consensus.cc:738] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:20.983539 1236 raft_consensus.cc:2747] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:20.983556 1170 consensus_queue.cc:260] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.985193 775 raft_consensus.cc:3058] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:20.987700 1170 ts_tablet_manager.cc:1428] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250905 19:52:20.990464 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1e9b7458f96148b08962a9b82223187e (DEFAULT_TABLE table=default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.990918 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9b7458f96148b08962a9b82223187e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.992018 740 tablet_service.cc:1468] Processing CreateTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.992636 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d38452d3a0641ffa4af5d935f8ad9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:20.993595 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1e9b7458f96148b08962a9b82223187e (DEFAULT_TABLE table=default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.993888 1184 ts_tablet_manager.cc:1918] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:20.996917 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4d6f2e44855d4312b1c0367145e3cee8 (DEFAULT_TABLE table=default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:20.997354 1169 tablet_bootstrap.cc:654] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:20.999665 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a8a69461d6db47e298856ec5bc896165" } new_table_name: "default.Table_5f244d1892624f63a48f35d92d075b5e" modify_external_catalogs: true
I20250905 19:52:21.002130 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c5d8d96a9bb34b5ea9b36388f41faf80" } new_table_name: "default.Table_63b0eb8b21074d33a20cc13f66b38ad9" modify_external_catalogs: true
I20250905 19:52:21.002152 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f6e643730c10465cb8585cb48c1c5e04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.002727 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6e643730c10465cb8585cb48c1c5e04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.005117 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 8092952d755848ccaba3a2ad94b66dad (DEFAULT_TABLE table=default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.007702 1184 log.cc:1199] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:21.008457 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.009869 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9b7458f96148b08962a9b82223187e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.012213 1170 tablet_bootstrap.cc:492] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.013727 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b129adef13534aacba0c176d41a4fca2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.014304 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b129adef13534aacba0c176d41a4fca2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.014739 1184 ts_tablet_manager.cc:1939] T 01c20eea1b024d7f93f18f0c089efc42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:21.017397 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:21.018142 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_07c414d9fb004a4086d372ed79538be1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.018697 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_07c414d9fb004a4086d372ed79538be1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.021965 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet f48f9f789b6e46ec871a0fd007770dfc
I20250905 19:52:21.023089 1673 raft_consensus.cc:2802] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:21.023257 1673 raft_consensus.cc:695] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:21.023308 893 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:20.988924 889 tablet_service.cc:1468] Processing CreateTablet for tablet 4d6f2e44855d4312b1c0367145e3cee8 (DEFAULT_TABLE table=default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.019908 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d136ca0885e44597a9298e4b7679fdf6" } modify_external_catalogs: true
I20250905 19:52:21.023908 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6f2e44855d4312b1c0367145e3cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.023715 1673 consensus_queue.cc:237] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:20.990702 891 tablet_service.cc:1468] Processing CreateTablet for tablet 1e9b7458f96148b08962a9b82223187e (DEFAULT_TABLE table=default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.025642 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1e9b7458f96148b08962a9b82223187e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.027881 1664 sys_catalog.cc:263] Verifying existing consensus state
I20250905 19:52:21.004120 886 tablet_service.cc:1468] Processing CreateTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.028240 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d38452d3a0641ffa4af5d935f8ad9f0. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.028512 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 45494.7us
I20250905 19:52:21.029088 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a3a50526bb604a218649b24cbf65a65e (DEFAULT_TABLE table=default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.029597 1236 raft_consensus.cc:491] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.030833 1185 ts_tablet_manager.cc:1918] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.030961 1185 log.cc:1199] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e6f837fb7293474b8ad357f52e42e42e
I20250905 19:52:21.031055 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8092952d755848ccaba3a2ad94b66dad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.031327 1185 ts_tablet_manager.cc:1939] T e6f837fb7293474b8ad357f52e42e42e P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:21.006376 878 tablet_service.cc:1468] Processing CreateTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (DEFAULT_TABLE table=default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.031817 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37894e39e6f2417fbfcd6d43f1ea2b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.032979 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (DEFAULT_TABLE table=default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.033289 1033 tablet_service.cc:1468] Processing CreateTablet for tablet e44e88a0df444fe5b156a8fdf3199507 (DEFAULT_TABLE table=default.Table_d34934e0d3f4442fa41d83040f628e19 [id=c96257f6e492466b88ad6da8f4abeb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.033653 1236 raft_consensus.cc:513] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.035812 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Bootstrap starting.
I20250905 19:52:21.009595 883 tablet_service.cc:1468] Processing CreateTablet for tablet e44e88a0df444fe5b156a8fdf3199507 (DEFAULT_TABLE table=default.Table_d34934e0d3f4442fa41d83040f628e19 [id=c96257f6e492466b88ad6da8f4abeb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.036247 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44e88a0df444fe5b156a8fdf3199507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.036276 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.037698 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8d38452d3a0641ffa4af5d935f8ad9f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.010269 885 tablet_service.cc:1468] Processing CreateTablet for tablet a3a50526bb604a218649b24cbf65a65e (DEFAULT_TABLE table=default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.039119 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a50526bb604a218649b24cbf65a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.040431 1185 tablet_replica.cc:331] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:21.046692 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e6f837fb7293474b8ad357f52e42e42e (table default.Table_1fd86ae5263542ed92214968ec16ccf0 [id=02ab99d2ef0f499db0d3625ede6c6e6e]) successfully deleted
I20250905 19:52:21.046700 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d136ca0885e44597a9298e4b7679fdf6" } modify_external_catalogs: true
I20250905 19:52:21.047423 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a50526bb604a218649b24cbf65a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.048702 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 01c20eea1b024d7f93f18f0c089efc42 (table default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]) successfully deleted
I20250905 19:52:21.021168 1171 tablet_bootstrap.cc:492] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:21.051172 454 catalog_manager.cc:5582] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:21.051628 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 2f581403ec924fb0948400df41ed8aab (table default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]). Expected version 1 got 0
I20250905 19:52:21.021696 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:21.051929 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_3f876cdc8a6e4380a81a2b4d49cfb138 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 1 got 0
I20250905 19:52:21.052646 454 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_f6b3a5016b7045bdbd914ad7ee173c25 [id=fe9667d9d9cf4b0e99651b827e50e4ac]). Expected version 1 got 0
I20250905 19:52:21.047572 1236 leader_election.cc:290] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:21.052198 1394 raft_consensus.cc:491] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.053575 1184 tablet_replica.cc:331] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:21.053416 1651 consensus_queue.cc:1035] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:21.054364 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44e88a0df444fe5b156a8fdf3199507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.055922 1394 raft_consensus.cc:513] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.056365 1675 tablet.cc:1722] T be73873da994402e84fff6aa2ded7525 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:21.059361 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2ed1bc09c94526bc9662f2d4d21f8f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:21.059468 1184 raft_consensus.cc:2241] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.061401 1394 leader_election.cc:290] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:21.062235 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet fbab0209a48d4ad8b2f2dec0562e9598
I20250905 19:52:21.062326 1184 raft_consensus.cc:2270] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.063014 1184 ts_tablet_manager.cc:1905] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:21.063858 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea6df8634a314ceabcf3fd004bf7e241 consensus state has no leader
I20250905 19:52:21.065994 891 tablet_service.cc:1515] Processing DeleteTablet for tablet fbab0209a48d4ad8b2f2dec0562e9598 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:21.066382 986 leader_election.cc:343] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.070053 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet fbab0209a48d4ad8b2f2dec0562e9598 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.070253 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37894e39e6f2417fbfcd6d43f1ea2b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.022063 1653 raft_consensus.cc:491] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.022238 1672 raft_consensus.cc:491] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.022782 1654 leader_election.cc:290] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.016650 888 tablet_service.cc:1468] Processing CreateTablet for tablet f2d929ec75d34cddad5286d8e37fcefb (DEFAULT_TABLE table=default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.040864 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:21.072820 741 tablet_service.cc:1468] Processing CreateTablet for tablet f2d929ec75d34cddad5286d8e37fcefb (DEFAULT_TABLE table=default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.074594 1185 raft_consensus.cc:2241] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.075270 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f2d929ec75d34cddad5286d8e37fcefb (DEFAULT_TABLE table=default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.075824 1171 ts_tablet_manager.cc:1397] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.166s user 0.001s sys 0.005s
W20250905 19:52:21.075778 986 leader_election.cc:343] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.076594 754 tablet_service.cc:1468] Processing CreateTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (DEFAULT_TABLE table=default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.077104 755 tablet_service.cc:1468] Processing CreateTablet for tablet a3a50526bb604a218649b24cbf65a65e (DEFAULT_TABLE table=default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.077608 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d929ec75d34cddad5286d8e37fcefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.077582 891 tablet_service.cc:1468] Processing CreateTablet for tablet f48f9f789b6e46ec871a0fd007770dfc (DEFAULT_TABLE table=default.Table_2af22804e18e430ea738862d0bade226 [id=82e44ec5bc0e43e2ab11552c4127f225]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.077592 880 tablet_service.cc:1468] Processing CreateTablet for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1 (DEFAULT_TABLE table=default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.078461 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1 (DEFAULT_TABLE table=default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.078963 750 tablet_service.cc:1468] Processing CreateTablet for tablet e44e88a0df444fe5b156a8fdf3199507 (DEFAULT_TABLE table=default.Table_d34934e0d3f4442fa41d83040f628e19 [id=c96257f6e492466b88ad6da8f4abeb84]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.079562 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d929ec75d34cddad5286d8e37fcefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.080418 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f2d929ec75d34cddad5286d8e37fcefb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.077088 1672 raft_consensus.cc:513] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.084496 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48f9f789b6e46ec871a0fd007770dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.084929 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.085700 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.088128 1185 raft_consensus.cc:2270] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.091405 1679 tablet.cc:1722] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:21.092006 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.092339 1185 ts_tablet_manager.cc:1905] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.098960 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9d6daaa0fe43414299c032d1f4d9bb6f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.099491 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.099699 747 tablet_service.cc:1515] Processing DeleteTablet for tablet fbab0209a48d4ad8b2f2dec0562e9598 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.100319 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d6f2e44855d4312b1c0367145e3cee8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.100409 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3a50526bb604a218649b24cbf65a65e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.102504 740 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.107283 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:21.108222 753 tablet_service.cc:1468] Processing CreateTablet for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1 (DEFAULT_TABLE table=default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.108282 1019 tablet_service.cc:1468] Processing CreateTablet for tablet f48f9f789b6e46ec871a0fd007770dfc (DEFAULT_TABLE table=default.Table_2af22804e18e430ea738862d0bade226 [id=82e44ec5bc0e43e2ab11552c4127f225]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.109769 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.111606 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37894e39e6f2417fbfcd6d43f1ea2b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.112340 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_9c5296ef0f424c5591eacb019ed29c7d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.113174 747 tablet_service.cc:1468] Processing CreateTablet for tablet f48f9f789b6e46ec871a0fd007770dfc (DEFAULT_TABLE table=default.Table_2af22804e18e430ea738862d0bade226 [id=82e44ec5bc0e43e2ab11552c4127f225]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:21.113405 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9c5296ef0f424c5591eacb019ed29c7d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.114792 455 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fe588d1872d847a7b9cd61b07c6f7abb" } new_table_name: "default.Table_708e16c265ea413f86dbd083d7b9dcde" modify_external_catalogs: true
I20250905 19:52:21.115672 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e44e88a0df444fe5b156a8fdf3199507. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.115713 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ae712f1c45694ed9b3b777066df1d465"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.116185 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f7f048cf13e2425186594ed068f5cb10"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.116667 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f7f048cf13e2425186594ed068f5cb10 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.111330 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9477afb5b47c46d4b24128a6c4bdabc6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:21.117453 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ae712f1c45694ed9b3b777066df1d465 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.119886 1170 tablet_bootstrap.cc:654] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.112437 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2ed1bc09c94526bc9662f2d4d21f8f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.122475 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.130709 821 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:21.134953 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48f9f789b6e46ec871a0fd007770dfc. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.112120 984 leader_election.cc:343] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.146833 984 leader_election.cc:304] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.149922 1236 raft_consensus.cc:2747] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.118808 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 6221d7d058944bdb9aa730791cf2417b (DEFAULT_TABLE table=default.Table_7a422c26de1b49dfa1ef67add2fc4fe8 [id=58e21ae57fda437e81365a58cfaa0fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.125733 1394 raft_consensus.cc:491] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.152853 1394 raft_consensus.cc:513] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.153473 1394 leader_election.cc:290] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:21.119171 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 0d93a328d4934e5294ec66f739eb240b (DEFAULT_TABLE table=default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.139498 1032 tablet_service.cc:1468] Processing CreateTablet for tablet d6b44c00d3b54af6b0752575e6898839 (DEFAULT_TABLE table=default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.140393 1169 tablet_bootstrap.cc:492] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:21.158612 1169 ts_tablet_manager.cc:1397] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.282s user 0.009s sys 0.000s
I20250905 19:52:21.159544 1169 raft_consensus.cc:357] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.159816 1169 raft_consensus.cc:383] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.159942 1169 raft_consensus.cc:738] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.160238 1169 consensus_queue.cc:260] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.160954 1169 ts_tablet_manager.cc:1428] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:21.161291 1169 tablet_bootstrap.cc:492] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:21.108624 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f48f9f789b6e46ec871a0fd007770dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.164103 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6221d7d058944bdb9aa730791cf2417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.167793 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d93a328d4934e5294ec66f739eb240b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.172992 1672 leader_election.cc:290] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.173189 1584 raft_consensus.cc:491] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.173343 1584 raft_consensus.cc:513] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.173981 1584 leader_election.cc:290] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.174149 1681 tablet.cc:1722] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:21.175069 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:21.176966 986 leader_election.cc:343] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.084034 1653 raft_consensus.cc:513] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.178282 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b44c00d3b54af6b0752575e6898839. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.181485 883 tablet_service.cc:1468] Processing CreateTablet for tablet 24bcd21f19ba465aa550102e0fccb674 (DEFAULT_TABLE table=default.Table_80258bf6ac33401fa32c1e2792e1d731 [id=c7b558dcabda400d9ba6e1850e0a61eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.181970 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24bcd21f19ba465aa550102e0fccb674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.182600 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:21.184890 848 consensus_peers.cc:489] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:21.185245 740 tablet_service.cc:1468] Processing CreateTablet for tablet 6221d7d058944bdb9aa730791cf2417b (DEFAULT_TABLE table=default.Table_7a422c26de1b49dfa1ef67add2fc4fe8 [id=58e21ae57fda437e81365a58cfaa0fb1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:21.188876 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 86541.3us
W20250905 19:52:21.189280 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d296428b70f9411190b6c408d81639f2 consensus state has no leader
I20250905 19:52:21.192418 1584 raft_consensus.cc:491] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.200017 1169 tablet_bootstrap.cc:654] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.204448 1171 raft_consensus.cc:357] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.204739 1171 raft_consensus.cc:383] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.210233 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 24bcd21f19ba465aa550102e0fccb674 (DEFAULT_TABLE table=default.Table_80258bf6ac33401fa32c1e2792e1d731 [id=c7b558dcabda400d9ba6e1850e0a61eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.215619 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "087172093fff4c41980dc2daeabb07c3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.217587 1171 raft_consensus.cc:738] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.216367 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6221d7d058944bdb9aa730791cf2417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.217921 885 tablet_service.cc:1468] Processing CreateTablet for tablet 0d93a328d4934e5294ec66f739eb240b (DEFAULT_TABLE table=default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.216010 1186 ts_tablet_manager.cc:1918] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.218701 1584 raft_consensus.cc:513] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.219373 1584 leader_election.cc:290] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.220023 893 tablet_service.cc:1468] Processing CreateTablet for tablet 6221d7d058944bdb9aa730791cf2417b (DEFAULT_TABLE table=default.Table_7a422c26de1b49dfa1ef67add2fc4fe8 [id=58e21ae57fda437e81365a58cfaa0fb1]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:21.220058 984 leader_election.cc:343] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.220216 984 leader_election.cc:304] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.220045 886 tablet_service.cc:1468] Processing CreateTablet for tablet d6b44c00d3b54af6b0752575e6898839 (DEFAULT_TABLE table=default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.221840 1171 consensus_queue.cc:260] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.221455 738 tablet_service.cc:1468] Processing CreateTablet for tablet d6b44c00d3b54af6b0752575e6898839 (DEFAULT_TABLE table=default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.222434 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.227034 1688 raft_consensus.cc:2747] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.227690 1184 ts_tablet_manager.cc:1918] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.228513 1184 log.cc:1199] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/262bcd1cfc994bb58e999b141b9bc7ea
I20250905 19:52:21.237083 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b44c00d3b54af6b0752575e6898839. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.243449 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_45e2499e36e8452fbe6d4d39b5adbe1d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.244444 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_37d80cf1db464cff9b77c0aff7a595c2" } new_table_name: "default.Table_dca507a43eee443493aaa2d0072f91b4" modify_external_catalogs: true
I20250905 19:52:21.245770 448 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet 310fc51e45a74b3598e746fbe3f05036
I20250905 19:52:21.246048 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c7c4cd0bb1b14ead83f2942518f4c30f" } modify_external_catalogs: true
I20250905 19:52:21.246196 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c7c4cd0bb1b14ead83f2942518f4c30f" } modify_external_catalogs: true
I20250905 19:52:21.239462 744 tablet_service.cc:1468] Processing CreateTablet for tablet 0d93a328d4934e5294ec66f739eb240b (DEFAULT_TABLE table=default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.248575 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c72c8c0956b54b06985f31edc378cb66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.249209 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c72c8c0956b54b06985f31edc378cb66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.250939 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_869aaac5cd2744e3b23de6c1060391a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.251389 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_869aaac5cd2744e3b23de6c1060391a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.252722 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c815e30da2544766834b52eaefab63c4" } modify_external_catalogs: true
I20250905 19:52:21.252923 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c815e30da2544766834b52eaefab63c4" } modify_external_catalogs: true
I20250905 19:52:21.253724 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_29345948e72a432f9d41b63ca0c45110" } modify_external_catalogs: true
I20250905 19:52:21.253916 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_29345948e72a432f9d41b63ca0c45110" } modify_external_catalogs: true
I20250905 19:52:21.253659 755 tablet_service.cc:1468] Processing CreateTablet for tablet 24bcd21f19ba465aa550102e0fccb674 (DEFAULT_TABLE table=default.Table_80258bf6ac33401fa32c1e2792e1d731 [id=c7b558dcabda400d9ba6e1850e0a61eb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.256551 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.257068 1186 log.cc:1199] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a884eca0e54a44dc920114e61a199689
I20250905 19:52:21.260440 1653 leader_election.cc:290] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.260715 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d93a328d4934e5294ec66f739eb240b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.261771 452 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 2 got 0
I20250905 19:52:21.261934 1600 raft_consensus.cc:491] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.262018 452 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_d6e9e58d90324cd2a7110d05c7588160 [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 2 got 0
I20250905 19:52:21.262082 1600 raft_consensus.cc:513] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:21.262935 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_45e2499e36e8452fbe6d4d39b5adbe1d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:21.263051 455 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 0fb8fe93061848a0b055979bad10f08b consensus state has no leader
I20250905 19:52:21.264153 1600 leader_election.cc:290] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:21.266899 848 leader_election.cc:343] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:21.267500 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 177950us
I20250905 19:52:21.267885 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_05541d16561646a2b97139e3503cf120" } new_table_name: "default.Table_801aaff2487e46b3b2e0b28b3c0f83dd" modify_external_catalogs: true
I20250905 19:52:21.270193 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6221d7d058944bdb9aa730791cf2417b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.271059 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_4ec0491dbe8841049cceaeee7c721211" } modify_external_catalogs: true
I20250905 19:52:21.271242 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_4ec0491dbe8841049cceaeee7c721211" } modify_external_catalogs: true
I20250905 19:52:21.272188 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_3e73d212c07541f5afb91c19b19745f3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.272701 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e73d212c07541f5afb91c19b19745f3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.275528 1171 ts_tablet_manager.cc:1428] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.150s user 0.000s sys 0.003s
W20250905 19:52:21.281194 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 4edfa9a656554e02b6649e8430c215fb consensus state has no leader
I20250905 19:52:21.283959 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d6b44c00d3b54af6b0752575e6898839. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.288406 1185 ts_tablet_manager.cc:1918] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.288555 1185 log.cc:1199] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/7c5807c1132747e6af79d93e7496d500
I20250905 19:52:21.290896 959 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:21.291211 1171 tablet_bootstrap.cc:492] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:21.297745 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0d93a328d4934e5294ec66f739eb240b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.298676 712 leader_election.cc:343] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 9d6daaa0fe43414299c032d1f4d9bb6f
I20250905 19:52:21.301057 1185 ts_tablet_manager.cc:1939] T 7c5807c1132747e6af79d93e7496d500 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:21.301667 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24bcd21f19ba465aa550102e0fccb674. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.303058 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 24bcd21f19ba465aa550102e0fccb674. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.301513 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 74440.5us
I20250905 19:52:21.308202 1664 log.cc:826] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:21.309471 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8a32f742eaf473e9f6f9bf214f53679" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.311329 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c76077e4adf45a7adc736b0e69032eb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.314579 1679 tablet.cc:1722] T be73873da994402e84fff6aa2ded7525 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:21.315614 1184 ts_tablet_manager.cc:1939] T 262bcd1cfc994bb58e999b141b9bc7ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:21.317606 1184 tablet_replica.cc:331] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:21.317655 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 262bcd1cfc994bb58e999b141b9bc7ea (table default.Table_7df10c45b04b4316ad24e8e2d141de9b [id=8f60da23a0864f92ba7c7ffcfa65277f]) successfully deleted
W20250905 19:52:21.321202 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6e89d23c4cb64f568b25f92a278b9d5e consensus state has no leader
I20250905 19:52:21.321556 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d6e9e58d90324cd2a7110d05c7588160" } new_table_name: "default.Table_196c4c640c614829bd5dcbd02150bc6a" modify_external_catalogs: true
I20250905 19:52:21.323637 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
W20250905 19:52:21.324760 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05ec559d0f83445892f8285f9bdeb2c8 consensus state has no leader
W20250905 19:52:21.326777 849 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:21.333600 883 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:21.336040 984 leader_election.cc:343] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.339430 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.339881 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "087172093fff4c41980dc2daeabb07c3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:21.339838 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:21.340613 1050 raft_consensus.cc:2466] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:21.340618 1053 raft_consensus.cc:2466] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:21.341054 984 leader_election.cc:304] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.341387 1236 raft_consensus.cc:2747] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.342981 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 7c5807c1132747e6af79d93e7496d500 (table default.Table_f3bcb1e148584db3b4336af7b8028437 [id=cd492ea5b4b4434b960ba37fcf479450]) successfully deleted
W20250905 19:52:21.343235 848 leader_election.cc:343] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:21.343482 848 leader_election.cc:343] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.340945 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c76077e4adf45a7adc736b0e69032eb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:21.344501 1047 raft_consensus.cc:2466] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:21.340677 1184 raft_consensus.cc:2241] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.344861 1184 raft_consensus.cc:2270] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.336336 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9477afb5b47c46d4b24128a6c4bdabc6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:21.345649 1048 raft_consensus.cc:3058] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.347012 1184 ts_tablet_manager.cc:1905] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:21.348371 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 114804us
I20250905 19:52:21.348495 738 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:21.350229 711 leader_election.cc:343] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9d6daaa0fe43414299c032d1f4d9bb6f
I20250905 19:52:21.350385 711 leader_election.cc:304] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:21.351230 1600 raft_consensus.cc:2747] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.352906 1185 tablet_replica.cc:331] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:21.353775 849 leader_election.cc:304] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.354318 849 leader_election.cc:304] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.354631 1653 raft_consensus.cc:2802] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:21.354753 1653 raft_consensus.cc:491] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:21.354854 1653 raft_consensus.cc:3058] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.357960 1651 raft_consensus.cc:2802] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:21.363508 1169 tablet_bootstrap.cc:492] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:21.363773 1186 ts_tablet_manager.cc:1939] T a884eca0e54a44dc920114e61a199689 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:21.364830 1186 tablet_replica.cc:331] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:21.364826 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a884eca0e54a44dc920114e61a199689 (table default.Table_5ce48afe3ba94518bd44699d763ac7a9 [id=8c67898a816046e084be67f08f5edb71]) successfully deleted
I20250905 19:52:21.365087 1186 raft_consensus.cc:2241] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.365250 1186 pending_rounds.cc:70] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
W20250905 19:52:21.365201 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd793bce71e344f18bb177a53ab1bfc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd793bce71e344f18bb177a53ab1bfc1 already in progress: creating tablet
I20250905 19:52:21.365296 1186 pending_rounds.cc:77] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 2 } timestamp: 7197089549079810048 op_type: ALTER_SCHEMA_OP alter_schema_request { tablet_id: "2c884954e5dc4445a12252953593d0bc" schema { columns { id: 10 name: "key" type: INT32 is_key: true is_nullable: false encoding: AUTO_ENCODING compression: DEFAULT_COMPRESSION cfile_block_size: 0 immutable: false } } schema_version: 1 new_table_name: "default.Table_b42374e796034b1cbfc8659587477d3c" dest_uuid: "835daf30e4324782848290be181ae78f" new_extra_config { } }
I20250905 19:52:21.365698 1186 raft_consensus.cc:2270] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:21.365784 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 36f966bf512d4cbe9ae406386db31533 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 36f966bf512d4cbe9ae406386db31533 already in progress: creating tablet
I20250905 19:52:21.366070 1169 ts_tablet_manager.cc:1397] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.202s user 0.026s sys 0.024s
I20250905 19:52:21.367772 1169 raft_consensus.cc:357] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.367987 1688 raft_consensus.cc:491] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.368216 1169 raft_consensus.cc:383] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.368296 1169 raft_consensus.cc:738] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.368667 1048 raft_consensus.cc:2466] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:21.369122 848 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.369084 1169 consensus_queue.cc:260] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.369632 1169 ts_tablet_manager.cc:1428] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:21.370220 1169 tablet_bootstrap.cc:492] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:21.372347 1688 raft_consensus.cc:513] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.372704 1236 raft_consensus.cc:491] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.372835 1236 raft_consensus.cc:513] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.373874 1236 leader_election.cc:290] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:21.373980 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "302053e1611b4e22a9a2e7b60ae68f27" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:21.374142 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "302053e1611b4e22a9a2e7b60ae68f27" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:21.376755 986 leader_election.cc:343] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.380548 1688 leader_election.cc:290] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:21.380718 848 leader_election.cc:304] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.380905 1679 tablet.cc:1722] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:21.382022 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.383837 1185 raft_consensus.cc:2241] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:21.383935 984 leader_election.cc:343] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.384076 1185 raft_consensus.cc:2270] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.384168 984 leader_election.cc:304] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.384699 1185 ts_tablet_manager.cc:1905] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.385330 1169 tablet_bootstrap.cc:654] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.385911 1651 raft_consensus.cc:491] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:21.386114 1651 raft_consensus.cc:3058] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:21.386173 984 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.386884 1688 raft_consensus.cc:2747] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.388628 1186 ts_tablet_manager.cc:1905] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.389412 1651 raft_consensus.cc:513] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.390213 1653 raft_consensus.cc:513] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.391271 1653 leader_election.cc:290] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.391618 849 leader_election.cc:304] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:21.391853 848 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.393697 1653 raft_consensus.cc:2802] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:21.393790 1653 raft_consensus.cc:491] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:21.393890 1653 raft_consensus.cc:3058] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.393798 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_80258bf6ac33401fa32c1e2792e1d731" } new_table_name: "default.Table_65d1d80076a24f24a1c5284f9e90ec10" modify_external_catalogs: true
I20250905 19:52:21.395715 849 leader_election.cc:304] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.396963 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:21.397886 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "087172093fff4c41980dc2daeabb07c3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:21.396440 848 consensus_peers.cc:489] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:21.398727 848 leader_election.cc:343] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:21.398918 848 leader_election.cc:343] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.399811 1586 raft_consensus.cc:2802] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:21.400053 1586 raft_consensus.cc:695] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:21.400336 1586 consensus_queue.cc:237] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.401043 1184 ts_tablet_manager.cc:1918] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250905 19:52:21.402074 848 consensus_peers.cc:489] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:21.403671 1171 tablet_bootstrap.cc:654] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.403713 1658 raft_consensus.cc:2747] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.403918 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:21.405488 1494 raft_consensus.cc:491] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.405685 1494 raft_consensus.cc:513] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.406311 1494 leader_election.cc:290] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.407639 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:21.408424 848 leader_election.cc:343] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:21.413394 848 consensus_peers.cc:489] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:21.432967 1170 tablet_bootstrap.cc:492] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
W20250905 19:52:21.441363 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 24bcd21f19ba465aa550102e0fccb674 consensus state has no leader
I20250905 19:52:21.450613 448 catalog_manager.cc:5582] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:21.451107 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]). Expected version 2 got 0
I20250905 19:52:21.451314 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:21.451416 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 2 got 1
W20250905 19:52:21.451530 986 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.451592 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:21.451723 1184 log.cc:1199] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:21.451788 448 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_d6e9e58d90324cd2a7110d05c7588160 [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 2 got 0
I20250905 19:52:21.452143 1184 ts_tablet_manager.cc:1939] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:21.452906 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:21.453315 1184 tablet_replica.cc:331] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:21.453513 1184 raft_consensus.cc:2241] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.454311 1184 raft_consensus.cc:2270] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.454684 452 master_service.cc:948] ReplaceTablet: received request to replace tablet 2683a36998674f739b41d2a57339954c from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:21.454851 1184 ts_tablet_manager.cc:1905] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.457383 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_65b1531f2ba945118e5fdcfb5603e704"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.457957 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_65b1531f2ba945118e5fdcfb5603e704 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.459131 893 tablet_service.cc:1468] Processing CreateTablet for tablet 52c99a76802b481c98f77dbd90de7cbb (DEFAULT_TABLE table=default.Table_29345948e72a432f9d41b63ca0c45110 [id=7ae4e86b2cb6498f90c5a0f9c662a82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.459538 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c99a76802b481c98f77dbd90de7cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.460237 986 leader_election.cc:304] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.460526 1098 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:21.460690 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:21.460606 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b71a53bad9454675a843d031c9f776bf"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.461087 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b71a53bad9454675a843d031c9f776bf in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.462271 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:21.462293 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_93a9ab7d352a4a3da35219aa6f20f432" } modify_external_catalogs: true
I20250905 19:52:21.462487 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_93a9ab7d352a4a3da35219aa6f20f432" } modify_external_catalogs: true
I20250905 19:52:21.462677 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.463057 1653 raft_consensus.cc:513] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.463359 1049 raft_consensus.cc:3058] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.463791 1653 leader_election.cc:290] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:21.464865 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:21.465698 1697 log.cc:927] Time spent T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250905 19:52:21.467561 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e20d0ff772494c13b5e9aa353a1da157"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.468065 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e20d0ff772494c13b5e9aa353a1da157 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.468434 1185 ts_tablet_manager.cc:1918] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.468575 1185 log.cc:1199] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:21.468953 1185 ts_tablet_manager.cc:1939] T 43111a3512fb4fc0bdf4ee245b16695b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:21.469280 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 54628965b82e470d811eff95ffc56c5f
I20250905 19:52:21.469414 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4ec178bc65804a6fb36bb46ecfe8378f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.469879 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4ec178bc65804a6fb36bb46ecfe8378f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.469980 450 master_service.cc:948] ReplaceTablet: received request to replace tablet a8a32f742eaf473e9f6f9bf214f53679 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:21.467585 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_3ee66cc5b45647de889e10775569ae66"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.471222 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ee66cc5b45647de889e10775569ae66 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.480314 450 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f6b3a5016b7045bdbd914ad7ee173c25" } new_table_name: "default.Table_d24bc38d75eb490880b8769aa303daeb" modify_external_catalogs: true
I20250905 19:52:21.481530 1688 raft_consensus.cc:2747] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:21.481712 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 65403.8us
I20250905 19:52:21.482555 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 68f2ccd8a7a64b89b44f2d6767ec3466 (table default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]) successfully deleted
I20250905 19:52:21.483315 1185 tablet_replica.cc:331] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:21.483541 1185 raft_consensus.cc:2241] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.483719 1185 raft_consensus.cc:2270] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.484078 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.484217 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 43111a3512fb4fc0bdf4ee245b16695b (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]) successfully deleted
I20250905 19:52:21.484242 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.484289 1185 ts_tablet_manager.cc:1905] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.474563 1651 leader_election.cc:290] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.479439 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:21.489215 1049 raft_consensus.cc:2466] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:21.492079 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:21.433825 1186 ts_tablet_manager.cc:1918] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:21.457568 738 tablet_service.cc:1468] Processing CreateTablet for tablet 52c99a76802b481c98f77dbd90de7cbb (DEFAULT_TABLE table=default.Table_29345948e72a432f9d41b63ca0c45110 [id=7ae4e86b2cb6498f90c5a0f9c662a82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.495069 1170 ts_tablet_manager.cc:1397] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.491s user 0.011s sys 0.005s
I20250905 19:52:21.495221 886 tablet_service.cc:1468] Processing CreateTablet for tablet 040e65fdb1104b07b953044f9b6b538a (DEFAULT_TABLE table=default.Table_4e5dec3aaf854f29aeda797c8cd3d811 [id=ec8ef6059b414e5189d80735b0cde61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.495421 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c99a76802b481c98f77dbd90de7cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.495671 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 040e65fdb1104b07b953044f9b6b538a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.464346 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.497447 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 8e0c478efe044ad0867e47f49374a66b (DEFAULT_TABLE table=default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.497825 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0c478efe044ad0867e47f49374a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.499326 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]). Expected version 2 got 0
I20250905 19:52:21.499513 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_196c4c640c614829bd5dcbd02150bc6a [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 3 got 0
I20250905 19:52:21.499687 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_dca507a43eee443493aaa2d0072f91b4 [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 4 got 0
I20250905 19:52:21.500329 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a1b4989f17454aa58c05727b797c2009 (DEFAULT_TABLE table=default.Table_ee3eaf9eb84d435187c73a394308861b [id=8225e2fcccbf436db5bfa9e05ba88f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.501246 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b4989f17454aa58c05727b797c2009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.502056 1019 tablet_service.cc:1468] Processing CreateTablet for tablet a311f9b943d143a9a52d7771ce7186d7 (DEFAULT_TABLE table=default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.502383 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a311f9b943d143a9a52d7771ce7186d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.465130 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c76077e4adf45a7adc736b0e69032eb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:21.504313 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.470782 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.494091 740 tablet_service.cc:1468] Processing CreateTablet for tablet 040e65fdb1104b07b953044f9b6b538a (DEFAULT_TABLE table=default.Table_4e5dec3aaf854f29aeda797c8cd3d811 [id=ec8ef6059b414e5189d80735b0cde61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.494199 753 tablet_service.cc:1468] Processing CreateTablet for tablet a1b4989f17454aa58c05727b797c2009 (DEFAULT_TABLE table=default.Table_ee3eaf9eb84d435187c73a394308861b [id=8225e2fcccbf436db5bfa9e05ba88f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.504606 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda (DEFAULT_TABLE table=default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.504720 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 040e65fdb1104b07b953044f9b6b538a (DEFAULT_TABLE table=default.Table_4e5dec3aaf854f29aeda797c8cd3d811 [id=ec8ef6059b414e5189d80735b0cde61a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.494372 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.494733 754 tablet_service.cc:1468] Processing CreateTablet for tablet 8e0c478efe044ad0867e47f49374a66b (DEFAULT_TABLE table=default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.504959 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.505766 883 tablet_service.cc:1468] Processing CreateTablet for tablet a1b4989f17454aa58c05727b797c2009 (DEFAULT_TABLE table=default.Table_ee3eaf9eb84d435187c73a394308861b [id=8225e2fcccbf436db5bfa9e05ba88f83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.506348 888 tablet_service.cc:1468] Processing CreateTablet for tablet a311f9b943d143a9a52d7771ce7186d7 (DEFAULT_TABLE table=default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.506847 880 tablet_service.cc:1468] Processing CreateTablet for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda (DEFAULT_TABLE table=default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.507261 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 040e65fdb1104b07b953044f9b6b538a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.510258 848 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.510782 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b4989f17454aa58c05727b797c2009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.511037 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 040e65fdb1104b07b953044f9b6b538a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.511320 747 tablet_service.cc:1468] Processing CreateTablet for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda (DEFAULT_TABLE table=default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.514087 889 tablet_service.cc:1468] Processing CreateTablet for tablet 9db56577a26e407b97fc33158c2c466c (DEFAULT_TABLE table=default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.514101 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 52c99a76802b481c98f77dbd90de7cbb (DEFAULT_TABLE table=default.Table_29345948e72a432f9d41b63ca0c45110 [id=7ae4e86b2cb6498f90c5a0f9c662a82f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.514451 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 52c99a76802b481c98f77dbd90de7cbb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.516677 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9db56577a26e407b97fc33158c2c466c (DEFAULT_TABLE table=default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.517040 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 53ac46ce05694bc2b53e195638be9e23 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:21.517125 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db56577a26e407b97fc33158c2c466c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.519908 451 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 2 got 1
I20250905 19:52:21.521726 1170 raft_consensus.cc:357] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.522392 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.523661 755 tablet_service.cc:1468] Processing CreateTablet for tablet a311f9b943d143a9a52d7771ce7186d7 (DEFAULT_TABLE table=default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.523931 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a1b4989f17454aa58c05727b797c2009. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.524277 744 tablet_service.cc:1468] Processing CreateTablet for tablet 9db56577a26e407b97fc33158c2c466c (DEFAULT_TABLE table=default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.526823 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_bc7323553e4a435a800e253b20df7321"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.527007 741 tablet_service.cc:1468] Processing CreateTablet for tablet 9193487b4ab74dafa76a386ddcaf674b (DEFAULT_TABLE table=default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:21.527459 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc7323553e4a435a800e253b20df7321 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.527479 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0c478efe044ad0867e47f49374a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.528412 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9193487b4ab74dafa76a386ddcaf674b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.529029 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db56577a26e407b97fc33158c2c466c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.529817 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_5f244d1892624f63a48f35d92d075b5e" } new_table_name: "default.Table_55ee6ff3b1904fd98253e035e91cb732" modify_external_catalogs: true
I20250905 19:52:21.530164 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_2cde62a9512e4445b5d0eb12119380f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.531172 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a311f9b943d143a9a52d7771ce7186d7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.531279 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2cde62a9512e4445b5d0eb12119380f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.532454 1186 log.cc:1199] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/2c884954e5dc4445a12252953593d0bc
I20250905 19:52:21.533016 1186 ts_tablet_manager.cc:1939] T 2c884954e5dc4445a12252953593d0bc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:21.533604 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 9193487b4ab74dafa76a386ddcaf674b (DEFAULT_TABLE table=default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.534199 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 2c884954e5dc4445a12252953593d0bc (table default.Table_b42374e796034b1cbfc8659587477d3c [id=26cde3c02a704bb4931f65f8dba00173]) successfully deleted
W20250905 19:52:21.534513 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet b13d282e1dc54d1faa2882c9b39d17cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b13d282e1dc54d1faa2882c9b39d17cb already in progress: creating tablet
I20250905 19:52:21.534765 1186 tablet_replica.cc:331] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:21.534780 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9193487b4ab74dafa76a386ddcaf674b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.534751 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2d84653c9e7a4857a7580fa85f8e4732 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2d84653c9e7a4857a7580fa85f8e4732 already in progress: creating tablet
I20250905 19:52:21.534969 1186 raft_consensus.cc:2241] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.535127 1186 raft_consensus.cc:2270] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.535665 1186 ts_tablet_manager.cc:1905] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.536342 1170 raft_consensus.cc:383] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.536543 1170 raft_consensus.cc:738] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.537096 1169 tablet_bootstrap.cc:492] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:21.536955 1170 consensus_queue.cc:260] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.537953 1170 ts_tablet_manager.cc:1428] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.043s user 0.000s sys 0.003s
I20250905 19:52:21.537276 1169 ts_tablet_manager.cc:1397] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.167s user 0.005s sys 0.009s
I20250905 19:52:21.542104 1169 raft_consensus.cc:357] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.542284 1169 raft_consensus.cc:383] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.542395 1169 raft_consensus.cc:738] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.542373 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4c76077e4adf45a7adc736b0e69032eb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:21.543008 1047 raft_consensus.cc:3058] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.543015 1169 consensus_queue.cc:260] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.545806 1700 tablet.cc:1722] T be73873da994402e84fff6aa2ded7525 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:21.546366 1047 raft_consensus.cc:2466] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:21.547953 1169 ts_tablet_manager.cc:1428] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:21.548143 1049 raft_consensus.cc:1273] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Refusing update from remote peer 835daf30e4324782848290be181ae78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:21.552987 1169 tablet_bootstrap.cc:492] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:21.554839 1169 tablet_bootstrap.cc:654] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.561872 447 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 8edf4b54974c427a855eee5792d75292
I20250905 19:52:21.563272 447 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 2 got 0
I20250905 19:52:21.564003 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "087172093fff4c41980dc2daeabb07c3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:21.564205 1049 raft_consensus.cc:3058] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.564500 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_71d2ee92e2ba427fb82bb9120ea13ad5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.565001 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.566649 1049 raft_consensus.cc:2466] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:21.567880 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_68450db54c5641a1a249cdcf5fa2d5ec"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:21.568440 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_68450db54c5641a1a249cdcf5fa2d5ec in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.573022 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:21.573295 1170 tablet_bootstrap.cc:492] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.574244 1169 tablet_bootstrap.cc:492] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:21.574913 1169 ts_tablet_manager.cc:1397] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.003s
I20250905 19:52:21.578176 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.578886 1169 raft_consensus.cc:357] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.580090 1169 raft_consensus.cc:383] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.580550 740 tablet_service.cc:1468] Processing CreateTablet for tablet 3fe2b75a3e59437c885abc18614f1da2 (DEFAULT_TABLE table=default.Table_51169c758cd646cd91f3bda396598e10 [id=d92c116575544c1c8d98befb5a420348]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.581000 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fe2b75a3e59437c885abc18614f1da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.580720 1169 raft_consensus.cc:738] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.575026 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:21.582505 1044 raft_consensus.cc:2466] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:21.582333 1169 consensus_queue.cc:260] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.585026 1184 ts_tablet_manager.cc:1918] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.585152 1184 log.cc:1199] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/74a7e7d9874e4d26bda3ab24a6e24adc
I20250905 19:52:21.585495 1184 ts_tablet_manager.cc:1939] T 74a7e7d9874e4d26bda3ab24a6e24adc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:21.585675 741 tablet_service.cc:1468] Processing CreateTablet for tablet c14dc571b586405f8fd1eaa452998978 (DEFAULT_TABLE table=default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.586408 1184 tablet_replica.cc:331] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:21.586534 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c14dc571b586405f8fd1eaa452998978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.586650 1184 raft_consensus.cc:2241] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.586947 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 74a7e7d9874e4d26bda3ab24a6e24adc (table default.Table_aebe460a114e490ab8dfed41377a96b8 [id=f03c03ae24744485b3ac71752d70f7d8]) successfully deleted
I20250905 19:52:21.593989 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 3fe2b75a3e59437c885abc18614f1da2 (DEFAULT_TABLE table=default.Table_51169c758cd646cd91f3bda396598e10 [id=d92c116575544c1c8d98befb5a420348]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.595088 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fe2b75a3e59437c885abc18614f1da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.595804 453 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 2 got 0
I20250905 19:52:21.596637 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_88ff02c5dce2458784f09b64cee688f1" } modify_external_catalogs: true
I20250905 19:52:21.596835 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_88ff02c5dce2458784f09b64cee688f1" } modify_external_catalogs: true
I20250905 19:52:21.597476 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c72c8c0956b54b06985f31edc378cb66" } new_table_name: "default.Table_6212ca6f887b4190bdf0b6f455c0bc3f" modify_external_catalogs: true
I20250905 19:52:21.598343 454 master_service.cc:948] ReplaceTablet: received request to replace tablet 8d38452d3a0641ffa4af5d935f8ad9f0 from {username='slave'} at 127.0.0.1:52586
W20250905 19:52:21.598887 450 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f732ae0627c4da5b1691cde9b4d5562 consensus state has no leader
I20250905 19:52:21.599193 450 master_service.cc:948] ReplaceTablet: received request to replace tablet 2c884954e5dc4445a12252953593d0bc from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:21.599718 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422] alter complete (version 5)
W20250905 19:52:21.600397 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea6df8634a314ceabcf3fd004bf7e241 consensus state has no leader
I20250905 19:52:21.603555 1021 tablet_service.cc:1468] Processing CreateTablet for tablet ad0760dfee934aacbd4ab1fb81e116c7 (DEFAULT_TABLE table=default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.603411 1022 tablet_service.cc:1468] Processing CreateTablet for tablet db098a883a61474fb06cb166041ee3e4 (DEFAULT_TABLE table=default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.604032 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0760dfee934aacbd4ab1fb81e116c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.604362 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db098a883a61474fb06cb166041ee3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.605659 1023 tablet_service.cc:1468] Processing CreateTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.605757 1169 ts_tablet_manager.cc:1428] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.030s user 0.002s sys 0.002s
I20250905 19:52:21.606087 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ff206366ad4344b4df5fc6a5584f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.607868 1033 tablet_service.cc:1468] Processing CreateTablet for tablet c4b31443b0f840749e12435e197a204a (DEFAULT_TABLE table=default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.608672 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 7f34bdf651d1412690039dc6ea1095d0 (DEFAULT_TABLE table=default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.611030 1186 ts_tablet_manager.cc:1918] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:21.611132 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.611663 1186 log.cc:1199] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9d6daaa0fe43414299c032d1f4d9bb6f
I20250905 19:52:21.612835 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f34bdf651d1412690039dc6ea1095d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.614513 1169 tablet_bootstrap.cc:492] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:21.616267 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b31443b0f840749e12435e197a204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.616902 743 tablet_service.cc:1468] Processing CreateTablet for tablet 7f34bdf651d1412690039dc6ea1095d0 (DEFAULT_TABLE table=default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.617276 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f34bdf651d1412690039dc6ea1095d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.620337 754 tablet_service.cc:1468] Processing CreateTablet for tablet db098a883a61474fb06cb166041ee3e4 (DEFAULT_TABLE table=default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.620682 753 tablet_service.cc:1468] Processing CreateTablet for tablet d011d1ec8f634386872e7ab257de3187 (DEFAULT_TABLE table=default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.621199 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db098a883a61474fb06cb166041ee3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.622781 750 tablet_service.cc:1468] Processing CreateTablet for tablet c4b31443b0f840749e12435e197a204a (DEFAULT_TABLE table=default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.623168 738 tablet_service.cc:1468] Processing CreateTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.623929 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d011d1ec8f634386872e7ab257de3187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.624958 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ff206366ad4344b4df5fc6a5584f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.624421 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:21.628041 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b31443b0f840749e12435e197a204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.628821 1186 ts_tablet_manager.cc:1939] T 9d6daaa0fe43414299c032d1f4d9bb6f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:21.629302 747 tablet_service.cc:1468] Processing CreateTablet for tablet ad0760dfee934aacbd4ab1fb81e116c7 (DEFAULT_TABLE table=default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.632629 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.633193 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:21.633715 1032 tablet_service.cc:1468] Processing CreateTablet for tablet d011d1ec8f634386872e7ab257de3187 (DEFAULT_TABLE table=default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.634668 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0760dfee934aacbd4ab1fb81e116c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.638780 1186 tablet_replica.cc:331] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:21.639531 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.639868 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9db56577a26e407b97fc33158c2c466c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.639984 1651 consensus_queue.cc:1035] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.077s
I20250905 19:52:21.640659 849 leader_election.cc:304] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.641536 849 leader_election.cc:304] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.642059 849 leader_election.cc:304] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.642429 849 leader_election.cc:304] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.642696 1685 raft_consensus.cc:2802] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:21.643098 1685 raft_consensus.cc:695] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:21.643287 1673 raft_consensus.cc:2802] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:21.643427 1685 consensus_queue.cc:237] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.644155 1586 consensus_queue.cc:1035] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:52:21.644590 1586 raft_consensus.cc:2802] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:21.644740 1586 raft_consensus.cc:695] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:21.639032 774 raft_consensus.cc:3058] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.644959 1586 consensus_queue.cc:237] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.645861 1658 raft_consensus.cc:2802] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:21.645963 1658 raft_consensus.cc:491] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:21.646049 1658 raft_consensus.cc:3058] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.648140 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9d6daaa0fe43414299c032d1f4d9bb6f (table default.Table_363859b13b284790a71ced21040081e4 [id=3d169e77e70f4a059cdf41f2f3a70171]) successfully deleted
I20250905 19:52:21.648659 1658 raft_consensus.cc:513] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.649446 1658 leader_election.cc:290] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.650454 1186 raft_consensus.cc:2241] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:21.650574 1675 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:21.650933 1186 raft_consensus.cc:2270] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:21.651204 1681 tablet.cc:1722] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:21.652228 1186 ts_tablet_manager.cc:1905] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.655364 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ca7a00374cab4386a886afd1d3a90342" } new_table_name: "default.Table_70be9db9097645b2b762cf0d447c6032" modify_external_catalogs: true
I20250905 19:52:21.655895 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e" } modify_external_catalogs: true
I20250905 19:52:21.656237 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e" } modify_external_catalogs: true
I20250905 19:52:21.660602 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Bootstrap replayed 1/1 log segments. Stats: ops{read=15 overwritten=0 applied=15 ignored=0} inserts{seen=23 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20250905 19:52:21.661293 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_e93ef86856c54ccdae803a15856cbc68" } new_table_name: "default.Table_25d956aeed244b4d89f84d8f43086c55" modify_external_catalogs: true
I20250905 19:52:21.661969 1664 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Bootstrap complete.
I20250905 19:52:21.672166 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a311f9b943d143a9a52d7771ce7186d7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.677351 1700 tablet.cc:1722] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:21.680905 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_929cfcb2535e452ebb48483e593faa7a" } modify_external_catalogs: true
I20250905 19:52:21.681178 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_929cfcb2535e452ebb48483e593faa7a" } modify_external_catalogs: true
I20250905 19:52:21.683283 1682 tablet.cc:1722] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:21.687081 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c8afd57df21440b5923aef4dd927ae45"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:21.687440 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:21.687711 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c8afd57df21440b5923aef4dd927ae45 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:21.691205 1664 raft_consensus.cc:357] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:21.696465 879 tablet_service.cc:1468] Processing CreateTablet for tablet 3fe2b75a3e59437c885abc18614f1da2 (DEFAULT_TABLE table=default.Table_51169c758cd646cd91f3bda396598e10 [id=d92c116575544c1c8d98befb5a420348]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.697085 1664 raft_consensus.cc:738] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e38333af29db4bf6af41ee5a25d6701e, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.698945 1664 consensus_queue.cc:260] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 15, Last appended: 2.15, Last appended by leader: 15, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:21.700956 1170 tablet_bootstrap.cc:654] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.703836 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:21.704380 885 tablet_service.cc:1468] Processing CreateTablet for tablet 8e0c478efe044ad0867e47f49374a66b (DEFAULT_TABLE table=default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.707841 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3fe2b75a3e59437c885abc18614f1da2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.700309 891 tablet_service.cc:1468] Processing CreateTablet for tablet 9193487b4ab74dafa76a386ddcaf674b (DEFAULT_TABLE table=default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:21.710917 848 leader_election.cc:343] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.712944 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9193487b4ab74dafa76a386ddcaf674b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.714737 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:21.717248 1709 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:21.720721 1709 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:21.719842 1664 sys_catalog.cc:564] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:52:21.722764 878 tablet_service.cc:1468] Processing CreateTablet for tablet c14dc571b586405f8fd1eaa452998978 (DEFAULT_TABLE table=default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.731102 890 tablet_service.cc:1468] Processing CreateTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a (DEFAULT_TABLE table=default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.732432 887 tablet_service.cc:1468] Processing CreateTablet for tablet ad0760dfee934aacbd4ab1fb81e116c7 (DEFAULT_TABLE table=default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.734287 882 tablet_service.cc:1468] Processing CreateTablet for tablet db098a883a61474fb06cb166041ee3e4 (DEFAULT_TABLE table=default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.734815 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c14dc571b586405f8fd1eaa452998978. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.735550 876 tablet_service.cc:1468] Processing CreateTablet for tablet c4b31443b0f840749e12435e197a204a (DEFAULT_TABLE table=default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.736141 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db098a883a61474fb06cb166041ee3e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.736860 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a2ff206366ad4344b4df5fc6a5584f8a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.737680 893 tablet_service.cc:1468] Processing CreateTablet for tablet d011d1ec8f634386872e7ab257de3187 (DEFAULT_TABLE table=default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.738098 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d011d1ec8f634386872e7ab257de3187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.738592 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad0760dfee934aacbd4ab1fb81e116c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.739852 877 tablet_service.cc:1468] Processing CreateTablet for tablet 7f34bdf651d1412690039dc6ea1095d0 (DEFAULT_TABLE table=default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.742971 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4b31443b0f840749e12435e197a204a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.745623 1186 ts_tablet_manager.cc:1918] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:21.747164 1170 tablet_bootstrap.cc:492] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:21.748208 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f34bdf651d1412690039dc6ea1095d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.747946 1186 log.cc:1199] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/fbab0209a48d4ad8b2f2dec0562e9598
I20250905 19:52:21.748200 1170 ts_tablet_manager.cc:1397] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.210s user 0.005s sys 0.006s
I20250905 19:52:21.749462 1170 raft_consensus.cc:357] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.749764 1170 raft_consensus.cc:383] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.749887 1170 raft_consensus.cc:738] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.750202 1170 consensus_queue.cc:260] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.750901 1170 ts_tablet_manager.cc:1428] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:21.751250 1170 tablet_bootstrap.cc:492] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.753134 1170 tablet_bootstrap.cc:654] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.756469 1186 ts_tablet_manager.cc:1939] T fbab0209a48d4ad8b2f2dec0562e9598 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:21.758373 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]) successfully deleted
W20250905 19:52:21.759214 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet f48f9f789b6e46ec871a0fd007770dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f48f9f789b6e46ec871a0fd007770dfc already in progress: creating tablet
W20250905 19:52:21.760259 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e already in progress: creating tablet
W20250905 19:52:21.761147 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 06850faa25de4b3d83f6bf98ccd370f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06850faa25de4b3d83f6bf98ccd370f0 already in progress: creating tablet
W20250905 19:52:21.761828 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 54628965b82e470d811eff95ffc56c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54628965b82e470d811eff95ffc56c5f already in progress: creating tablet
W20250905 19:52:21.762483 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c99a76802b481c98f77dbd90de7cbb already in progress: creating tablet
W20250905 19:52:21.763942 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8edf4b54974c427a855eee5792d75292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8edf4b54974c427a855eee5792d75292 already in progress: creating tablet
I20250905 19:52:21.769831 1170 tablet_bootstrap.cc:492] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:21.770159 1170 ts_tablet_manager.cc:1397] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.004s
I20250905 19:52:21.773648 1170 raft_consensus.cc:357] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.774036 1170 raft_consensus.cc:383] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.774199 1170 raft_consensus.cc:738] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.774652 1170 consensus_queue.cc:260] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.775571 1170 ts_tablet_manager.cc:1428] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:21.776063 1170 tablet_bootstrap.cc:492] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.779024 1170 tablet_bootstrap.cc:654] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.796371 1170 tablet_bootstrap.cc:492] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:21.796734 1170 ts_tablet_manager.cc:1397] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.001s
I20250905 19:52:21.798527 1170 raft_consensus.cc:357] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.798950 1170 raft_consensus.cc:383] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.799185 1170 raft_consensus.cc:738] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.799629 1170 consensus_queue.cc:260] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.800529 1170 ts_tablet_manager.cc:1428] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250905 19:52:21.803632 1170 tablet_bootstrap.cc:492] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.805688 1170 tablet_bootstrap.cc:654] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.807163 1685 raft_consensus.cc:491] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.807343 1685 raft_consensus.cc:513] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.807991 1685 leader_election.cc:290] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.808614 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:21.810141 848 leader_election.cc:343] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.821223 1170 tablet_bootstrap.cc:492] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:21.821528 1170 ts_tablet_manager.cc:1397] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20250905 19:52:21.822471 1170 raft_consensus.cc:357] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.822777 1170 raft_consensus.cc:383] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.822927 1170 raft_consensus.cc:738] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.823249 1170 consensus_queue.cc:260] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.824095 1170 ts_tablet_manager.cc:1428] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:21.824450 1170 tablet_bootstrap.cc:492] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.830781 1170 tablet_bootstrap.cc:654] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.838155 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8e0c478efe044ad0867e47f49374a66b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.844236 1170 tablet_bootstrap.cc:492] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:21.844604 1170 ts_tablet_manager.cc:1397] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.020s user 0.006s sys 0.001s
I20250905 19:52:21.845754 1170 raft_consensus.cc:357] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.846077 1170 raft_consensus.cc:383] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.846228 1170 raft_consensus.cc:738] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.846652 1170 consensus_queue.cc:260] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.847332 1170 ts_tablet_manager.cc:1428] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:21.847721 1170 tablet_bootstrap.cc:492] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:21.856154 1170 tablet_bootstrap.cc:654] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.863713 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d011d1ec8f634386872e7ab257de3187. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:21.865116 1688 raft_consensus.cc:491] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.865324 1688 raft_consensus.cc:513] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.866314 1688 leader_election.cc:290] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:21.867611 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ec559d0f83445892f8285f9bdeb2c8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:21.868855 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ec559d0f83445892f8285f9bdeb2c8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.869968 1020 tablet_service.cc:1468] Processing CreateTablet for tablet c14dc571b586405f8fd1eaa452998978 (DEFAULT_TABLE table=default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:21.874601 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c14dc571b586405f8fd1eaa452998978. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:21.875334 986 leader_election.cc:343] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:21.876978 984 leader_election.cc:343] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.877306 984 leader_election.cc:304] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.882217 1688 raft_consensus.cc:2747] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.892289 1169 tablet_bootstrap.cc:654] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:21.896914 1688 raft_consensus.cc:491] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:21.897078 1688 raft_consensus.cc:513] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.898932 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28e403dc2f6471ebea1805fd843cfb1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:21.899096 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28e403dc2f6471ebea1805fd843cfb1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:21.900745 1688 leader_election.cc:290] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:21.901018 986 leader_election.cc:343] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:21.903371 984 leader_election.cc:343] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.903668 984 leader_election.cc:304] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.904016 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:21.905460 1048 raft_consensus.cc:3058] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.913825 1688 raft_consensus.cc:2747] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:21.916817 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:21.921411 1053 raft_consensus.cc:2466] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:21.922739 849 leader_election.cc:304] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.923399 1672 raft_consensus.cc:2802] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:21.923563 1672 raft_consensus.cc:491] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:21.923677 1672 raft_consensus.cc:3058] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:21.929284 1672 raft_consensus.cc:513] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.947094 1672 leader_election.cc:290] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:21.948136 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:21.943393 1728 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:21.950559 1048 raft_consensus.cc:2466] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:21.971100 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:21.971561 1048 raft_consensus.cc:3058] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:21.949162 848 leader_election.cc:343] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:21.955611 849 leader_election.cc:304] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:21.979914 1672 raft_consensus.cc:2802] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:21.980347 1672 raft_consensus.cc:695] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:21.980901 1672 consensus_queue.cc:237] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:21.995170 1171 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.591s user 0.101s sys 0.150s
W20250905 19:52:21.995493 1171 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.591s user 0.101s sys 0.150s
I20250905 19:52:21.996444 1171 tablet_bootstrap.cc:492] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:21.996865 1171 ts_tablet_manager.cc:1397] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.721s user 0.103s sys 0.153s
I20250905 19:52:21.997843 1171 raft_consensus.cc:357] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.998139 1171 raft_consensus.cc:383] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:21.998308 1171 raft_consensus.cc:738] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:21.998749 1171 consensus_queue.cc:260] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:21.999419 1171 ts_tablet_manager.cc:1428] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:21.999776 1171 tablet_bootstrap.cc:492] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:21.998778 848 consensus_peers.cc:489] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:22.002396 1171 tablet_bootstrap.cc:654] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:22.005963 1685 raft_consensus.cc:491] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:22.006171 1685 raft_consensus.cc:513] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.007015 1685 leader_election.cc:290] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.009362 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a41dbd8782460cb0d3c5f8bd59354f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:22.010079 848 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:22.010562 1171 tablet_bootstrap.cc:492] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:22.010692 1171 ts_tablet_manager.cc:1397] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20250905 19:52:22.011397 1171 raft_consensus.cc:357] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.011621 1171 raft_consensus.cc:383] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:22.011720 1171 raft_consensus.cc:738] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:22.012023 1171 consensus_queue.cc:260] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.012511 1171 ts_tablet_manager.cc:1428] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:22.012796 1171 tablet_bootstrap.cc:492] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:22.013513 775 raft_consensus.cc:3058] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:22.014652 1171 tablet_bootstrap.cc:654] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:22.024263 1171 tablet_bootstrap.cc:492] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:22.024454 1171 ts_tablet_manager.cc:1397] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20250905 19:52:22.025445 1171 raft_consensus.cc:357] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.025667 1171 raft_consensus.cc:383] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:22.025745 1171 raft_consensus.cc:738] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:22.026052 1171 consensus_queue.cc:260] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.026583 1171 ts_tablet_manager.cc:1428] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:22.026891 1171 tablet_bootstrap.cc:492] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:22.029311 1171 tablet_bootstrap.cc:654] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:22.029721 1184 raft_consensus.cc:2270] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:22.032469 1709 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: New leader c401c9f3020b4ae5931c566c6bcb1711. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:22.032722 1709 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: FOLLOWER
W20250905 19:52:22.033720 849 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:22.033915 849 leader_election.cc:304] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:22.036486 1683 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:22.036710 1683 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:22.037688 1685 raft_consensus.cc:2747] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:22.049453 1046 raft_consensus.cc:1273] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Refusing update from remote peer 835daf30e4324782848290be181ae78f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20250905 19:52:22.050001 1048 raft_consensus.cc:2466] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:22.051167 1685 consensus_queue.cc:1035] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:22.052829 849 leader_election.cc:304] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:22.053435 1685 raft_consensus.cc:2802] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:22.053615 1685 raft_consensus.cc:695] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:22.053864 1685 consensus_queue.cc:237] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.061441 1600 raft_consensus.cc:491] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:22.061713 1600 raft_consensus.cc:513] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.062597 1600 leader_election.cc:290] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:22.064599 712 leader_election.cc:343] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: e6f837fb7293474b8ad357f52e42e42e
W20250905 19:52:22.065485 711 leader_election.cc:343] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: e6f837fb7293474b8ad357f52e42e42e
I20250905 19:52:22.065624 711 leader_election.cc:304] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:22.066215 1600 raft_consensus.cc:2747] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:22.075940 1184 ts_tablet_manager.cc:1905] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:21.643287 1185 ts_tablet_manager.cc:1918] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:22.081905 1185 log.cc:1199] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:22.082544 1185 ts_tablet_manager.cc:1939] T a8a32f742eaf473e9f6f9bf214f53679 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:22.083307 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 303) took 1388 ms. Trace:
I20250905 19:52:22.083525 1185 rpcz_store.cc:276] 0905 19:52:20.694879 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.695017 (+ 138us) service_pool.cc:225] Handling call
0905 19:52:21.483224 (+788207us) ts_tablet_manager.cc:1145] Deleting tablet a8a32f742eaf473e9f6f9bf214f53679
0905 19:52:21.483234 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:21.487943 (+ 4709us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:21.643137 (+155194us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:22.083243 (+440106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":727844}
I20250905 19:52:22.085649 1185 tablet_replica.cc:331] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:22.089543 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6597c91bec4f4143ab3ed40afe53a35e has no consensus state
I20250905 19:52:22.090157 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:22.090634 454 catalog_manager.cc:6652] ReplaceTablet: tablet 8d38452d3a0641ffa4af5d935f8ad9f0 deleted and replaced by tablet 160a7814abf9423dbe965c252c20fa7f
I20250905 19:52:22.091032 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:22.091988 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d38452d3a0641ffa4af5d935f8ad9f0 already in progress: creating tablet
I20250905 19:52:22.092829 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 3de936d2309a411eabbd59f7b73ce7f3
W20250905 19:52:22.094867 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 03a12cf719f34096bb82f5846bbe3b2d consensus state has no leader
W20250905 19:52:22.095669 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 61667a379974487195f81043683b9a24 consensus state has no leader
I20250905 19:52:22.097904 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 2f581403ec924fb0948400df41ed8aab
I20250905 19:52:22.099575 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f581403ec924fb0948400df41ed8aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.099850 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.101161 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b740696e409f49ba9fc25fb046f7b6e8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.101233 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2f581403ec924fb0948400df41ed8aab because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2f581403ec924fb0948400df41ed8aab already in progress: creating tablet
I20250905 19:52:22.102545 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_bb2028f0f4b14f999945c905baf2bc71"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:22.102655 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8a59f0ac61434a5880164439529876f4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:22.103219 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a21b18f7481c4713823fadf4b164b7bf" } modify_external_catalogs: true
I20250905 19:52:22.103379 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_a21b18f7481c4713823fadf4b164b7bf" } modify_external_catalogs: true
I20250905 19:52:22.103772 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_571c61e5a382457cbe5deececb435567"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.104234 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3de936d2309a411eabbd59f7b73ce7f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3de936d2309a411eabbd59f7b73ce7f3 already in progress: creating tablet
W20250905 19:52:22.104504 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b740696e409f49ba9fc25fb046f7b6e8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:22.104907 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8a59f0ac61434a5880164439529876f4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.105716 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_0e8d769f23e9495cae82a20a516763f7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.106173 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0e8d769f23e9495cae82a20a516763f7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.107417 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e065514fff894d6c837edd542a939abd"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.107693 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_571c61e5a382457cbe5deececb435567 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:22.107877 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e065514fff894d6c837edd542a939abd in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.108139 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_65d1d80076a24f24a1c5284f9e90ec10" } modify_external_catalogs: true
I20250905 19:52:22.108335 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_65d1d80076a24f24a1c5284f9e90ec10" } modify_external_catalogs: true
W20250905 19:52:22.109004 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bb2028f0f4b14f999945c905baf2bc71 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.111698 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f581403ec924fb0948400df41ed8aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.112154 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.115047 1728 tablet.cc:1722] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:22.132194 1688 raft_consensus.cc:491] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:22.132488 1688 raft_consensus.cc:513] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.133421 1688 leader_election.cc:290] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:22.135568 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d8144bb4f14358af539b7aae326918" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:22.137038 984 leader_election.cc:343] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:22.138633 446 catalog_manager.cc:5582] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:22.139045 446 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:22.142673 1019 tablet_service.cc:1468] Processing CreateTablet for tablet cd5312852fe345d98039aeea241b7fe8 (DEFAULT_TABLE table=default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.143236 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5312852fe345d98039aeea241b7fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.145084 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6597c91bec4f4143ab3ed40afe53a35e (DEFAULT_TABLE table=default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.145514 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6597c91bec4f4143ab3ed40afe53a35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.148782 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 3b53259c71d5443ebb12827ee78038fe (DEFAULT_TABLE table=default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.148842 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 98e4e6d274574c0580315dcc2323c3d6 (DEFAULT_TABLE table=default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.149226 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e4e6d274574c0580315dcc2323c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.152220 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_8b6a52346ae4431c93ff61ccda0c25fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:22.152529 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b53259c71d5443ebb12827ee78038fe. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.152761 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b6a52346ae4431c93ff61ccda0c25fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.153921 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_4e5dec3aaf854f29aeda797c8cd3d811" } new_table_name: "default.Table_cae023e67f764978833e906977ad0dde" modify_external_catalogs: true
I20250905 19:52:22.156769 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 9f953a7490434bc7afa5510c33e79401 (DEFAULT_TABLE table=default.Table_f7f048cf13e2425186594ed068f5cb10 [id=3d53333f33bc4acd8572699247ce0e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.157146 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f953a7490434bc7afa5510c33e79401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.162082 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_196c4c640c614829bd5dcbd02150bc6a" } new_table_name: "default.Table_bfc0123a4cd9449ba4bcb59bf2e88ddb" modify_external_catalogs: true
I20250905 19:52:22.163885 1752 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c401c9f3020b4ae5931c566c6bcb1711)
I20250905 19:52:22.164101 1752 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:22.165892 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 3 candidate_status { last_received { term: 2 index: 480 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:22.167377 1752 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:22.171888 1742 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:22.172219 1742 catalog_manager.cc:1554] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: loading cluster ID for follower catalog manager: success
I20250905 19:52:22.180749 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 3609de83732d4ad8857372acfbb5c32a (DEFAULT_TABLE table=default.Table_f6e643730c10465cb8585cb48c1c5e04 [id=fa983711b6574ebeb9dfb2cca302bdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.181214 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3609de83732d4ad8857372acfbb5c32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.185112 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 55cecf59d3f24db88bb10b8e12c12f82 (DEFAULT_TABLE table=default.Table_2cde62a9512e4445b5d0eb12119380f1 [id=46715848277847218d557157e4dfc77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.185335 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6bb79e3126074736a23b128a8e14fab6 (DEFAULT_TABLE table=default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.185606 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55cecf59d3f24db88bb10b8e12c12f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.185814 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb79e3126074736a23b128a8e14fab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.186154 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 3c824f5591044c4198a51a25661f71e5 (DEFAULT_TABLE table=default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.186499 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c824f5591044c4198a51a25661f71e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.192301 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (DEFAULT_TABLE table=default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.192710 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdc9d9815864aa7bf5ca9a5c4485390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.194203 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 97d949c3c24e40fe8fefd3a5758d57ea (DEFAULT_TABLE table=default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.195210 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d949c3c24e40fe8fefd3a5758d57ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.200376 1742 catalog_manager.cc:1576] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: acquiring CA information for follower catalog manager: success
I20250905 19:52:22.204686 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 48be704b238e481089c727b3e1b96a2f (DEFAULT_TABLE table=default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.205235 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48be704b238e481089c727b3e1b96a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.212543 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 3 candidate_status { last_received { term: 2 index: 480 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:22.213955 512 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:22.214345 1752 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:52:22.208534 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 1d095804b51b4205a23ac3d3106312f2 (DEFAULT_TABLE table=default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.218714 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419 (DEFAULT_TABLE table=default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.219257 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.220628 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d095804b51b4205a23ac3d3106312f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.222631 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 32fa16972f2a42d39b338b833c185b28 (DEFAULT_TABLE table=default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.223080 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5302b79208494e48a5f2778c5f52ad01 (DEFAULT_TABLE table=default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.217541 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 95b32fff7d4745c79185a5d894ef3342 (DEFAULT_TABLE table=default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.221035 1020 tablet_service.cc:1468] Processing CreateTablet for tablet f034b8cb5d8943938cb2b58c33b45494 (DEFAULT_TABLE table=default.Table_bc7323553e4a435a800e253b20df7321 [id=9d1528c0d2b1414e9cd5b222124016df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.224184 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:22.224355 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32fa16972f2a42d39b338b833c185b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.225240 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:22.225100 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b32fff7d4745c79185a5d894ef3342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.228608 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5302b79208494e48a5f2778c5f52ad01. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.231523 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 040e65fdb1104b07b953044f9b6b538a consensus state has no leader
I20250905 19:52:22.232044 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 24bcd21f19ba465aa550102e0fccb674
I20250905 19:52:22.237588 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.238822 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.214141 1742 catalog_manager.cc:1604] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:52:22.244657 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f034b8cb5d8943938cb2b58c33b45494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.271409 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.272022 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:22.285928 442 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 24bcd21f19ba465aa550102e0fccb674 (table default.Table_65d1d80076a24f24a1c5284f9e90ec10 [id=c7b558dcabda400d9ba6e1850e0a61eb], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 24bcd21f19ba465aa550102e0fccb674 consensus state has no leader
I20250905 19:52:22.287806 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6e5d1701ba1440d7a446f7831e77d23a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.288470 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e5d1701ba1440d7a446f7831e77d23a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.290650 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f115b7b7862d49bda223a405a360e7b8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.291214 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f115b7b7862d49bda223a405a360e7b8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:22.299561 440 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:41317, user_credentials={real_user=slave}} blocked reactor thread for 181223us
I20250905 19:52:22.300915 753 tablet_service.cc:1468] Processing CreateTablet for tablet f034b8cb5d8943938cb2b58c33b45494 (DEFAULT_TABLE table=default.Table_bc7323553e4a435a800e253b20df7321 [id=9d1528c0d2b1414e9cd5b222124016df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.302157 747 tablet_service.cc:1468] Processing CreateTablet for tablet cd5312852fe345d98039aeea241b7fe8 (DEFAULT_TABLE table=default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.302925 754 tablet_service.cc:1468] Processing CreateTablet for tablet 48be704b238e481089c727b3e1b96a2f (DEFAULT_TABLE table=default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.303226 743 tablet_service.cc:1468] Processing CreateTablet for tablet 97d949c3c24e40fe8fefd3a5758d57ea (DEFAULT_TABLE table=default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.303635 750 tablet_service.cc:1468] Processing CreateTablet for tablet 6597c91bec4f4143ab3ed40afe53a35e (DEFAULT_TABLE table=default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.303826 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9554a085b43c42a98edcb684be28faaa" } new_table_name: "default.Table_175c78c56c0149e8919f8220e87e3d4b" modify_external_catalogs: true
I20250905 19:52:22.304028 738 tablet_service.cc:1468] Processing CreateTablet for tablet 3b53259c71d5443ebb12827ee78038fe (DEFAULT_TABLE table=default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.304701 755 tablet_service.cc:1468] Processing CreateTablet for tablet 95b32fff7d4745c79185a5d894ef3342 (DEFAULT_TABLE table=default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.304952 740 tablet_service.cc:1468] Processing CreateTablet for tablet 9f953a7490434bc7afa5510c33e79401 (DEFAULT_TABLE table=default.Table_f7f048cf13e2425186594ed068f5cb10 [id=3d53333f33bc4acd8572699247ce0e77]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.305388 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_63b0eb8b21074d33a20cc13f66b38ad9" } modify_external_catalogs: true
I20250905 19:52:22.305423 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1d095804b51b4205a23ac3d3106312f2 (DEFAULT_TABLE table=default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.305557 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_63b0eb8b21074d33a20cc13f66b38ad9" } modify_external_catalogs: true
I20250905 19:52:22.304698 741 tablet_service.cc:1468] Processing CreateTablet for tablet 98e4e6d274574c0580315dcc2323c3d6 (DEFAULT_TABLE table=default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.306103 739 tablet_service.cc:1468] Processing CreateTablet for tablet 3609de83732d4ad8857372acfbb5c32a (DEFAULT_TABLE table=default.Table_f6e643730c10465cb8585cb48c1c5e04 [id=fa983711b6574ebeb9dfb2cca302bdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.306676 746 tablet_service.cc:1468] Processing CreateTablet for tablet 55cecf59d3f24db88bb10b8e12c12f82 (DEFAULT_TABLE table=default.Table_2cde62a9512e4445b5d0eb12119380f1 [id=46715848277847218d557157e4dfc77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.307334 756 tablet_service.cc:1468] Processing CreateTablet for tablet 3c824f5591044c4198a51a25661f71e5 (DEFAULT_TABLE table=default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.307441 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f43b3348e3084812974970a576a6fd15" } modify_external_catalogs: true
I20250905 19:52:22.307595 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f43b3348e3084812974970a576a6fd15" } modify_external_catalogs: true
I20250905 19:52:22.308248 745 tablet_service.cc:1468] Processing CreateTablet for tablet 6bb79e3126074736a23b128a8e14fab6 (DEFAULT_TABLE table=default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.307854 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_79a0ddd169324b64b69f90b3d65bfa11" } new_table_name: "default.Table_849b4e364b3f4ff2b47a2347f84d6b0d" modify_external_catalogs: true
I20250905 19:52:22.309377 742 tablet_service.cc:1468] Processing CreateTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (DEFAULT_TABLE table=default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.309712 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_bc7323553e4a435a800e253b20df7321" } new_table_name: "default.Table_4c30c89397974e059c9e48f15077d515" modify_external_catalogs: true
I20250905 19:52:22.312422 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_015ff1997237469aa213edd41b3b6ea4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.312943 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_015ff1997237469aa213edd41b3b6ea4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.134552 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d8144bb4f14358af539b7aae326918" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:22.267349 1702 raft_consensus.cc:491] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:22.315595 882 consensus_meta.cc:287] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f: Time spent flushing consensus metadata: real 0.548s user 0.004s sys 0.001s
I20250905 19:52:22.323374 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:22.328328 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f581403ec924fb0948400df41ed8aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:22.328680 883 tablet_service.cc:1468] Processing CreateTablet for tablet 6597c91bec4f4143ab3ed40afe53a35e (DEFAULT_TABLE table=default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.328845 891 tablet_service.cc:1468] Processing CreateTablet for tablet 48be704b238e481089c727b3e1b96a2f (DEFAULT_TABLE table=default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:22.331776 698 kernel_stack_watchdog.cc:198] Thread 774 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/util/thread.cc:642 for 428ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb0da75e980 (unknown) at ??:0
@ 0x92cb07 __sanitizer::internal_sched_yield() at ??:?
@ 0x8921bd __interceptor_pthread_create at ??:?
@ 0x7fb0d72fafb1 kudu::Thread::StartThread() at ??:0
@ 0xa63f97 kudu::Thread::Create() at ??:?
@ 0x7fb0d731adfb kudu::ThreadPool::CreateThread() at ??:0
@ 0x7fb0d731681d kudu::ThreadPool::DoSubmit() at ??:0
@ 0x7fb0d731bb4f kudu::ThreadPool::Submit() at ??:0
@ 0x7fb0e41191e8 kudu::tablet::OpDriver::ApplyAsync() at ??:0
@ 0x7fb0e411a27f kudu::tablet::OpDriver::ReplicationFinished() at ??:0
@ 0x7fb0e40c652d kudu::tablet::TabletReplica::StartFollowerOp()::$_7::operator()() at ??:0
@ 0x7fb0e40c6252 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb0eaa16c0e std::function<>::operator()() at ??:0
@ 0x7fb0e2da709d kudu::consensus::ConsensusRound::NotifyReplicationFinished() at ??:0
@ 0x7fb0e2d31e7a kudu::consensus::PendingRounds::AdvanceCommittedIndex() at ??:0
@ 0x7fb0e2d7eaf6 kudu::consensus::RaftConsensus::UpdateReplica() at ??:0
I20250905 19:52:22.340644 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d296428b70f9411190b6c408d81639f2
I20250905 19:52:22.344153 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 56e7ab709fe04479a7b1f17dba2fd603 (DEFAULT_TABLE table=default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.350539 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e7ab709fe04479a7b1f17dba2fd603. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.370009 986 leader_election.cc:343] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:22.370190 986 leader_election.cc:304] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:22.370569 1688 raft_consensus.cc:2747] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:22.371399 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b (DEFAULT_TABLE table=default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.371838 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a59d7fb5dc54b479d965282b1c7d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.382175 1031 tablet_service.cc:1468] Processing CreateTablet for tablet fa5d34f9c0ee4d55ade61b30b0f4c293 (DEFAULT_TABLE table=default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.383252 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5d34f9c0ee4d55ade61b30b0f4c293. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.388782 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e7086c5bfa4644d08e4f55231b85acf4 (DEFAULT_TABLE table=default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.388677 1032 tablet_service.cc:1468] Processing CreateTablet for tablet d7155cf5abba4e2cbdce583406f94fad (DEFAULT_TABLE table=default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.392563 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7086c5bfa4644d08e4f55231b85acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.396287 1025 tablet_service.cc:1468] Processing CreateTablet for tablet fd1280fefc3e43db9928567a734644f1 (DEFAULT_TABLE table=default.Table_c8afd57df21440b5923aef4dd927ae45 [id=4447588b8d2443e5a561758689326e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.344208 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48be704b238e481089c727b3e1b96a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.414568 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb79e3126074736a23b128a8e14fab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.353032 1600 raft_consensus.cc:491] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:22.414762 1600 raft_consensus.cc:513] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.415483 1600 leader_election.cc:290] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.415684 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f953a7490434bc7afa5510c33e79401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.420096 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e4e6d274574c0580315dcc2323c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.420342 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3609de83732d4ad8857372acfbb5c32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.420562 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f034b8cb5d8943938cb2b58c33b45494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.420758 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55cecf59d3f24db88bb10b8e12c12f82. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.423466 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5312852fe345d98039aeea241b7fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.423985 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b53259c71d5443ebb12827ee78038fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.425926 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d949c3c24e40fe8fefd3a5758d57ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.426585 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdc9d9815864aa7bf5ca9a5c4485390. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.429277 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:22.430328 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c824f5591044c4198a51a25661f71e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.431245 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b32fff7d4745c79185a5d894ef3342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.432132 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6597c91bec4f4143ab3ed40afe53a35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.435791 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d095804b51b4205a23ac3d3106312f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.434346 875 tablet_service.cc:1468] Processing CreateTablet for tablet 3c824f5591044c4198a51a25661f71e5 (DEFAULT_TABLE table=default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.437300 1702 raft_consensus.cc:513] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.325675 884 tablet_service.cc:1468] Processing CreateTablet for tablet 3609de83732d4ad8857372acfbb5c32a (DEFAULT_TABLE table=default.Table_f6e643730c10465cb8585cb48c1c5e04 [id=fa983711b6574ebeb9dfb2cca302bdc1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.439462 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "01c20eea1b024d7f93f18f0c089efc42" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:22.440253 848 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:22.440676 880 tablet_service.cc:1468] Processing CreateTablet for tablet 3b53259c71d5443ebb12827ee78038fe (DEFAULT_TABLE table=default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.442662 886 tablet_service.cc:1468] Processing CreateTablet for tablet 9f953a7490434bc7afa5510c33e79401 (DEFAULT_TABLE table=default.Table_f7f048cf13e2425186594ed068f5cb10 [id=3d53333f33bc4acd8572699247ce0e77]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:22.455410 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc95a83ca872495894ef439cdda4e364 consensus state has no leader
W20250905 19:52:22.456905 454 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet f034b8cb5d8943938cb2b58c33b45494 consensus state has no leader
I20250905 19:52:22.457043 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_515c33e99f2c486abcf9e9333253193b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:22.457268 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c8afd57df21440b5923aef4dd927ae45" } modify_external_catalogs: true
I20250905 19:52:22.457446 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_c8afd57df21440b5923aef4dd927ae45" } modify_external_catalogs: true
I20250905 19:52:22.458781 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_572ff86c294b4e0faaed684f96c62536"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.459295 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_572ff86c294b4e0faaed684f96c62536 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.460220 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.460680 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.460812 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7155cf5abba4e2cbdce583406f94fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.462311 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ad054f6da70449ecad529f69c6aa9705"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.462760 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ad054f6da70449ecad529f69c6aa9705 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.463462 452 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_175c78c56c0149e8919f8220e87e3d4b" } modify_external_catalogs: true
I20250905 19:52:22.463632 452 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_175c78c56c0149e8919f8220e87e3d4b" } modify_external_catalogs: true
I20250905 19:52:22.464164 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1280fefc3e43db9928567a734644f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.464260 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_95826d2c4a88413cbb1afa1d780a8f0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.464697 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_95826d2c4a88413cbb1afa1d780a8f0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.464953 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_1c3606c3e6e0452b910b8b3ce847fb58"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.465425 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_1c3606c3e6e0452b910b8b3ce847fb58 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.465951 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b21bb0be9ae1435c8929de3f6b662d8c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.466404 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b21bb0be9ae1435c8929de3f6b662d8c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:22.457690 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_515c33e99f2c486abcf9e9333253193b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.475898 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet fd1280fefc3e43db9928567a734644f1
I20250905 19:52:22.477403 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3" } modify_external_catalogs: true
I20250905 19:52:22.477588 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3" } modify_external_catalogs: true
I20250905 19:52:22.478425 1029 tablet_service.cc:1468] Processing CreateTablet for tablet ae6fe4620dda4b1296416e01f8ffee29 (DEFAULT_TABLE table=default.Table_571c61e5a382457cbe5deececb435567 [id=a42dd4a1d931496f8786c21f55f1c87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.482456 740 tablet_service.cc:1468] Processing CreateTablet for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419 (DEFAULT_TABLE table=default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.482877 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.483815 452 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet cf2881d228964cd3bef040a06045d8d2
I20250905 19:52:22.485708 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.485956 751 tablet_service.cc:1468] Processing CreateTablet for tablet 5302b79208494e48a5f2778c5f52ad01 (DEFAULT_TABLE table=default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.489445 877 tablet_service.cc:1468] Processing CreateTablet for tablet f034b8cb5d8943938cb2b58c33b45494 (DEFAULT_TABLE table=default.Table_bc7323553e4a435a800e253b20df7321 [id=9d1528c0d2b1414e9cd5b222124016df]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.489811 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 29c2b030fafd44ad9e648b2527a4ad07
I20250905 19:52:22.506879 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.509660 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet cf2881d228964cd3bef040a06045d8d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.513505 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.514282 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.516671 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5302b79208494e48a5f2778c5f52ad01. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.524097 711 leader_election.cc:343] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:22.528225 737 tablet_service.cc:1468] Processing CreateTablet for tablet 56e7ab709fe04479a7b1f17dba2fd603 (DEFAULT_TABLE table=default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.528687 749 tablet_service.cc:1468] Processing CreateTablet for tablet 32fa16972f2a42d39b338b833c185b28 (DEFAULT_TABLE table=default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.530094 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:22.528527 774 thread.cc:641] apply [worker] (thread pool) Time spent creating pthread: real 0.860s user 0.083s sys 0.121s
I20250905 19:52:22.489238 890 tablet_service.cc:1468] Processing CreateTablet for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419 (DEFAULT_TABLE table=default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.535661 881 tablet_service.cc:1468] Processing CreateTablet for tablet 55cecf59d3f24db88bb10b8e12c12f82 (DEFAULT_TABLE table=default.Table_2cde62a9512e4445b5d0eb12119380f1 [id=46715848277847218d557157e4dfc77b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.535751 1685 consensus_queue.cc:1035] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.147s
I20250905 19:52:22.536042 876 tablet_service.cc:1468] Processing CreateTablet for tablet cd5312852fe345d98039aeea241b7fe8 (DEFAULT_TABLE table=default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.535691 889 tablet_service.cc:1468] Processing CreateTablet for tablet 98e4e6d274574c0580315dcc2323c3d6 (DEFAULT_TABLE table=default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.536417 1185 raft_consensus.cc:2241] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:22.536473 894 tablet_service.cc:1468] Processing CreateTablet for tablet 1d095804b51b4205a23ac3d3106312f2 (DEFAULT_TABLE table=default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.536552 887 tablet_service.cc:1468] Processing CreateTablet for tablet 97d949c3c24e40fe8fefd3a5758d57ea (DEFAULT_TABLE table=default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.536631 1185 raft_consensus.cc:2270] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:22.536878 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6597c91bec4f4143ab3ed40afe53a35e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.537118 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1d095804b51b4205a23ac3d3106312f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.537324 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97d949c3c24e40fe8fefd3a5758d57ea. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.537248 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "57f851a6eecc4a41806bdaa19cc5afab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:22.537600 879 tablet_service.cc:1468] Processing CreateTablet for tablet 6bb79e3126074736a23b128a8e14fab6 (DEFAULT_TABLE table=default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.537945 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bb79e3126074736a23b128a8e14fab6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.539101 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c824f5591044c4198a51a25661f71e5. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.540033 848 consensus_peers.cc:489] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:22.540148 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 86703us
W20250905 19:52:22.540539 848 consensus_peers.cc:489] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fbab0209a48d4ad8b2f2dec0562e9598. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:22.540822 1702 leader_election.cc:290] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.544200 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98e4e6d274574c0580315dcc2323c3d6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.545645 1185 ts_tablet_manager.cc:1905] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:22.548089 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 55cecf59d3f24db88bb10b8e12c12f82. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.549808 1584 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.005s sys 0.009s
W20250905 19:52:22.549906 1584 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.907s user 0.006s sys 0.009s
W20250905 19:52:22.550294 1673 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.907s user 0.012s sys 0.003s
W20250905 19:52:22.550379 1673 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.907s user 0.012s sys 0.003s
I20250905 19:52:22.550472 1673 raft_consensus.cc:695] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:22.550737 1673 consensus_queue.cc:237] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.552408 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cd5312852fe345d98039aeea241b7fe8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.556150 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3609de83732d4ad8857372acfbb5c32a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.556491 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f953a7490434bc7afa5510c33e79401. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.559396 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b53259c71d5443ebb12827ee78038fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.565184 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6d527034680144f190c70d035f183b04"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.565831 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6d527034680144f190c70d035f183b04 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.565955 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 48be704b238e481089c727b3e1b96a2f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.566236 447 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f115b7b7862d49bda223a405a360e7b8" } modify_external_catalogs: true
I20250905 19:52:22.566501 447 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f115b7b7862d49bda223a405a360e7b8" } modify_external_catalogs: true
I20250905 19:52:22.566730 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_dca507a43eee443493aaa2d0072f91b4" } new_table_name: "default.Table_45e1bab15bd84e51a1a749ebfd98eeab" modify_external_catalogs: true
I20250905 19:52:22.569272 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7b30c5a5c660490191d00c27224d516f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.569784 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7b30c5a5c660490191d00c27224d516f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.571516 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f7f048cf13e2425186594ed068f5cb10" } new_table_name: "default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20" modify_external_catalogs: true
I20250905 19:52:22.574349 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_826b7d0620fd47ccb449a28ee9469199" } new_table_name: "default.Table_f6f360749f78488fa03d88b2cfad1817" modify_external_catalogs: true
I20250905 19:52:22.582731 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32fa16972f2a42d39b338b833c185b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.587153 741 tablet_service.cc:1468] Processing CreateTablet for tablet fa5d34f9c0ee4d55ade61b30b0f4c293 (DEFAULT_TABLE table=default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.588798 748 tablet_service.cc:1468] Processing CreateTablet for tablet ae6fe4620dda4b1296416e01f8ffee29 (DEFAULT_TABLE table=default.Table_571c61e5a382457cbe5deececb435567 [id=a42dd4a1d931496f8786c21f55f1c87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.593510 892 tablet_service.cc:1468] Processing CreateTablet for tablet 95b32fff7d4745c79185a5d894ef3342 (DEFAULT_TABLE table=default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.596230 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 66df0522aa2842f68aa824571fa61b05
W20250905 19:52:22.591126 849 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:22.598861 888 tablet_service.cc:1468] Processing CreateTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (DEFAULT_TABLE table=default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.601197 1730 tablet.cc:1722] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:22.601594 1729 raft_consensus.cc:491] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:22.601759 1729 raft_consensus.cc:513] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.602006 1031 tablet_service.cc:1468] Processing CreateTablet for tablet be1315824fd749049a62af3e6a113b9a (DEFAULT_TABLE table=default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.602519 1729 leader_election.cc:290] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:22.602697 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae6fe4620dda4b1296416e01f8ffee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.603564 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1315824fd749049a62af3e6a113b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.604344 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 421b28d047124222ae9cbc603ec22cee (DEFAULT_TABLE table=default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.604724 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421b28d047124222ae9cbc603ec22cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.608045 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 (DEFAULT_TABLE table=default.Table_68450db54c5641a1a249cdcf5fa2d5ec [id=94f2531a27494c32bcd549a25287867f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.608448 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ebab2e3698f4f60a770bde9cc019600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.613555 1184 ts_tablet_manager.cc:1918] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:22.613696 1184 log.cc:1199] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:22.614017 1184 ts_tablet_manager.cc:1939] T 4c76077e4adf45a7adc736b0e69032eb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:22.614321 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 292) took 1989 ms. Trace:
I20250905 19:52:22.615381 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 2568b9f6f37b48efbe92cef4c19d5dfb (DEFAULT_TABLE table=default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.615808 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2568b9f6f37b48efbe92cef4c19d5dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.616454 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 66df0522aa2842f68aa824571fa61b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.617373 447 catalog_manager.cc:5864] Not sending DeleteTablet requests; no consensus state for tablet f536befbca72451db71f512f490908da
W20250905 19:52:22.617719 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 89a73e984cbb4268b563f1ca01f4c65d consensus state has no leader
W20250905 19:52:22.618134 453 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 05ec559d0f83445892f8285f9bdeb2c8 consensus state has no leader
W20250905 19:52:22.618599 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 9f953a7490434bc7afa5510c33e79401 consensus state has no leader
I20250905 19:52:22.620031 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:22.621106 753 tablet_service.cc:1468] Processing CreateTablet for tablet 160a7814abf9423dbe965c252c20fa7f (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.624794 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 160a7814abf9423dbe965c252c20fa7f (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.625252 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a7814abf9423dbe965c252c20fa7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.630877 744 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.636026 1600 raft_consensus.cc:491] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:22.641674 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:22.642045 1044 thread.cc:641] raft [worker] (thread pool) Time spent creating pthread: real 0.514s user 0.000s sys 0.008s
W20250905 19:52:22.642256 1044 thread.cc:608] raft [worker] (thread pool) Time spent starting thread: real 0.729s user 0.000s sys 0.008s
I20250905 19:52:22.644657 1169 tablet_bootstrap.cc:492] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:22.644987 1169 ts_tablet_manager.cc:1397] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 1.031s user 0.005s sys 0.012s
I20250905 19:52:22.614431 1184 rpcz_store.cc:276] 0905 19:52:20.624930 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.625009 (+ 79us) service_pool.cc:225] Handling call
0905 19:52:21.586339 (+961330us) ts_tablet_manager.cc:1145] Deleting tablet 4c76077e4adf45a7adc736b0e69032eb
0905 19:52:21.586350 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:22.495622 (+909272us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:22.613448 (+117826us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:22.614272 (+ 824us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":956129}
I20250905 19:52:22.646896 1169 raft_consensus.cc:357] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.648810 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 4c76077e4adf45a7adc736b0e69032eb (table default.Table_48447ad862474825a10962298c97650d [id=d467b5191bf744858924c0ecbbf2e53c]) successfully deleted
I20250905 19:52:22.649384 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f034b8cb5d8943938cb2b58c33b45494. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.650924 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 305) took 1964 ms. Trace:
I20250905 19:52:22.651022 1184 rpcz_store.cc:276] 0905 19:52:20.686185 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.688012 (+ 1827us) service_pool.cc:225] Handling call
0905 19:52:22.650164 (+1962152us) ts_tablet_manager.cc:1145] Deleting tablet baefaa1f042348c1874e306f1e5a8781
0905 19:52:22.650173 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:22.650284 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1960456}
W20250905 19:52:22.651584 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet baefaa1f042348c1874e306f1e5a8781 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet baefaa1f042348c1874e306f1e5a8781 already in progress: creating tablet
I20250905 19:52:22.653318 1184 tablet_replica.cc:331] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:22.653621 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ae83eea5f74e9187713e8b82c9a23b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:22.654213 986 leader_election.cc:343] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:22.657788 448 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7a422c26de1b49dfa1ef67add2fc4fe8" } new_table_name: "default.Table_21f0cf32be59451d987d978191525180" modify_external_catalogs: true
I20250905 19:52:22.658032 882 tablet_service.cc:1468] Processing CreateTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b (DEFAULT_TABLE table=default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.659788 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 95b32fff7d4745c79185a5d894ef3342. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.653541 1184 raft_consensus.cc:2241] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:22.660730 885 tablet_service.cc:1468] Processing CreateTablet for tablet 32fa16972f2a42d39b338b833c185b28 (DEFAULT_TABLE table=default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:22.666669 1737 log.cc:927] Time spent T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.080s user 0.000s sys 0.000s
W20250905 19:52:22.668972 1586 thread.cc:641] wal-append [worker] (thread pool) Time spent creating pthread: real 0.948s user 0.009s sys 0.008s
W20250905 19:52:22.669085 1586 thread.cc:608] wal-append [worker] (thread pool) Time spent starting thread: real 1.023s user 0.009s sys 0.008s
I20250905 19:52:22.669500 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d34934e0d3f4442fa41d83040f628e19" } new_table_name: "default.Table_f34aac6beca44bc3bf4326aa500aaf21" modify_external_catalogs: true
I20250905 19:52:22.670976 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_858dac3c36484123a1a9de7c20d0f141"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:22.671537 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_858dac3c36484123a1a9de7c20d0f141 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.673959 1708 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250905 19:52:22.674330 774 thread.cc:608] apply [worker] (thread pool) Time spent starting thread: real 0.863s user 0.083s sys 0.121s
I20250905 19:52:22.674361 752 tablet_service.cc:1468] Processing CreateTablet for tablet d7155cf5abba4e2cbdce583406f94fad (DEFAULT_TABLE table=default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.674641 774 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.104.2:59145 (request call id 101) took 1036 ms. Trace:
I20250905 19:52:22.674743 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e7ab709fe04479a7b1f17dba2fd603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.674794 774 rpcz_store.cc:276] 0905 19:52:21.638533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.638593 (+ 60us) service_pool.cc:225] Handling call
0905 19:52:21.638884 (+ 291us) raft_consensus.cc:1469] Updating replica for 3 ops
0905 19:52:21.648170 (+ 9286us) raft_consensus.cc:1532] Early marking committed up to index 0
0905 19:52:21.648175 (+ 5us) raft_consensus.cc:1537] Triggering prepare for 3 ops
0905 19:52:21.648375 (+ 200us) alter_schema_op.cc:128] START. Timestamp: 1757101940731953
0905 19:52:21.648918 (+ 543us) alter_schema_op.cc:128] START. Timestamp: 1757101941252969
0905 19:52:21.649611 (+ 693us) log.cc:844] Serialized 372 byte log entry
0905 19:52:21.667941 (+ 18330us) raft_consensus.cc:1643] Marking committed up to 2
0905 19:52:21.668210 (+ 269us) log.cc:844] Serialized 14 byte log entry
0905 19:52:22.674478 (+1006268us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:52:22.674513 (+ 35us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:52:22.674524 (+ 11us) raft_consensus.cc:1683] finished
0905 19:52:22.674526 (+ 2us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:52:22.674590 (+ 64us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:21.648504 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:21.648648 (+ 144us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:21.648656 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:21.648662 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:21.668664 (+ 20002us) op_driver.cc:464] REPLICATION: finished
0905 19:52:21.669662 (+ 998us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:21.677842 (+ 8180us) log.cc:844] Serialized 5 byte log entry
0905 19:52:21.741880 (+ 64038us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:21.742182 (+ 302us) log.cc:844] Serialized 14 byte log entry
0905 19:52:21.742241 (+ 59us) alter_schema_op.cc:73] Released schema lock
0905 19:52:21.742246 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Related trace 'op':
0905 19:52:21.649040 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:21.649132 (+ 92us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:21.742306 (+ 93174us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:21.742315 (+ 9us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
Metrics: {"prepare.queue_time_us":74327,"prepare.run_cpu_time_us":6,"prepare.run_wall_time_us":7,"raft.queue_time_us":129,"raft.run_cpu_time_us":137,"raft.run_wall_time_us":153,"thread_start_us":18139,"threads_started":1,"wal-append.queue_time_us":83396,"child_traces":[["op",{"apply.queue_time_us":911,"apply.run_cpu_time_us":4358,"apply.run_wall_time_us":72656,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":173,"prepare.run_wall_time_us":176,"replication_time_us":20185,"thread_start_us":1005659,"threads_started":1}],["op",{"prepare.queue_time_us":63,"prepare.run_cpu_time_us":11313,"prepare.run_wall_time_us":93290}]]}
I20250905 19:52:22.676932 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7155cf5abba4e2cbdce583406f94fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.679071 1184 raft_consensus.cc:2270] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:22.679781 1184 ts_tablet_manager.cc:1905] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:22.680253 1032 tablet_service.cc:1468] Processing CreateTablet for tablet d495b6ba92e84b588b4780c2896195c0 (DEFAULT_TABLE table=default.Table_ad054f6da70449ecad529f69c6aa9705 [id=c31b7e937b6d4adfb299ae8e46330d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.680491 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 (DEFAULT_TABLE table=default.Table_b21bb0be9ae1435c8929de3f6b662d8c [id=a9a1d004ba9f4415b2fb958c789d66ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.680651 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d495b6ba92e84b588b4780c2896195c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.682127 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae6fe4620dda4b1296416e01f8ffee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.684340 747 tablet_service.cc:1468] Processing CreateTablet for tablet fd1280fefc3e43db9928567a734644f1 (DEFAULT_TABLE table=default.Table_c8afd57df21440b5923aef4dd927ae45 [id=4447588b8d2443e5a561758689326e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.684479 1546 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" instance_seqno: 1757101932934030) as {username='slave'} at 127.0.104.1:44759; Asking this server to re-register.
I20250905 19:52:22.684738 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1280fefc3e43db9928567a734644f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.684957 1170 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.757s user 0.048s sys 0.116s
W20250905 19:52:22.685039 448 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6221d7d058944bdb9aa730791cf2417b consensus state has no leader
W20250905 19:52:22.685053 1170 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 0.827s user 0.048s sys 0.116s
I20250905 19:52:22.685370 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d5eb8bec074d2c8766fe7b159031e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.685421 742 tablet_service.cc:1468] Processing CreateTablet for tablet 2568b9f6f37b48efbe92cef4c19d5dfb (DEFAULT_TABLE table=default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.686100 1027 tablet_service.cc:1468] Processing CreateTablet for tablet e5fc2c2ba49444af939f5a975165362b (DEFAULT_TABLE table=default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.686579 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5fc2c2ba49444af939f5a975165362b. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.689999 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0267ac3e3f1a45978807737eca5d91ef because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0267ac3e3f1a45978807737eca5d91ef already in progress: creating tablet
W20250905 19:52:22.691042 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 24bcd21f19ba465aa550102e0fccb674 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 24bcd21f19ba465aa550102e0fccb674 already in progress: creating tablet
W20250905 19:52:22.691541 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 5bd9380e8ffc432da4a684934c2df68f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 5bd9380e8ffc432da4a684934c2df68f already in progress: creating tablet
I20250905 19:52:22.692993 755 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.693017 849 leader_election.cc:304] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:22.693871 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 213199us
I20250905 19:52:22.694000 1600 raft_consensus.cc:513] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.694674 1600 leader_election.cc:290] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.694965 1672 raft_consensus.cc:2747] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:22.696101 745 tablet_service.cc:1468] Processing CreateTablet for tablet e7086c5bfa4644d08e4f55231b85acf4 (DEFAULT_TABLE table=default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.697561 750 tablet_service.cc:1468] Processing CreateTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b (DEFAULT_TABLE table=default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.697741 1730 tablet.cc:1722] T 607c8a8848304491bbd4a7971f96322f P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250905 19:52:22.701818 711 leader_election.cc:343] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 77cbdda0e2d641bd8092f00925bb9a9c
I20250905 19:52:22.703315 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5d34f9c0ee4d55ade61b30b0f4c293. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.705919 887 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:22.707295 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2568b9f6f37b48efbe92cef4c19d5dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.709439 755 tablet_service.cc:1468] Processing CreateTablet for tablet e10b4e82c4bc429b9a42db39c03a5267 (DEFAULT_TABLE table=default.Table_1c3606c3e6e0452b910b8b3ce847fb58 [id=6a2d8d9aa80544c6b70397d8f061865b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.710944 744 tablet_service.cc:1515] Processing DeleteTablet for tablet cf2881d228964cd3bef040a06045d8d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:22.725577 848 consensus_peers.cc:489] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:22.726291 1584 raft_consensus.cc:491] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:23.210386 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 486255us
I20250905 19:52:23.210433 1584 raft_consensus.cc:513] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.211171 1584 leader_election.cc:290] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.729166 887 tablet_service.cc:1468] Processing CreateTablet for tablet e7086c5bfa4644d08e4f55231b85acf4 (DEFAULT_TABLE table=default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.751101 878 tablet_service.cc:1468] Processing CreateTablet for tablet 5302b79208494e48a5f2778c5f52ad01 (DEFAULT_TABLE table=default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.995996 1783 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c401c9f3020b4ae5931c566c6bcb1711)
I20250905 19:52:22.751897 891 tablet_service.cc:1468] Processing CreateTablet for tablet fa5d34f9c0ee4d55ade61b30b0f4c293 (DEFAULT_TABLE table=default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.213279 1783 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:23.213973 1783 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:22.828333 1546 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "835daf30e4324782848290be181ae78f" instance_seqno: 1757101933433043) as {username='slave'} at 127.0.104.2:40691; Asking this server to re-register.
I20250905 19:52:22.728439 1023 tablet_service.cc:1468] Processing CreateTablet for tablet ca646f7c162e41469f283657e72823b4 (DEFAULT_TABLE table=default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.214272 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36f966bf512d4cbe9ae406386db31533" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:23.214502 1045 raft_consensus.cc:2466] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:23.214622 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca646f7c162e41469f283657e72823b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.214936 849 leader_election.cc:304] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:23.215245 1703 raft_consensus.cc:2802] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:23.215329 1703 raft_consensus.cc:491] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:23.215421 1703 raft_consensus.cc:3058] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:22.714991 754 tablet_service.cc:1468] Processing CreateTablet for tablet 421b28d047124222ae9cbc603ec22cee (DEFAULT_TABLE table=default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.710493 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_cd6dae29e54e4735a5252b0638fcf5b7"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:23.218410 1703 raft_consensus.cc:513] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:23.218540 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cd6dae29e54e4735a5252b0638fcf5b7 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.219944 1703 leader_election.cc:290] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.752285 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:22.753723 880 tablet_service.cc:1468] Processing CreateTablet for tablet d7155cf5abba4e2cbdce583406f94fad (DEFAULT_TABLE table=default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:22.754807 1171 thread.cc:641] log-alloc [worker] (thread pool) Time spent creating pthread: real 0.612s user 0.047s sys 0.075s
I20250905 19:52:23.220973 894 tablet_service.cc:1515] Processing DeleteTablet for tablet cf2881d228964cd3bef040a06045d8d2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:23.221119 1171 thread.cc:608] log-alloc [worker] (thread pool) Time spent starting thread: real 1.191s user 0.047s sys 0.075s
I20250905 19:52:22.756152 875 tablet_service.cc:1468] Processing CreateTablet for tablet ae6fe4620dda4b1296416e01f8ffee29 (DEFAULT_TABLE table=default.Table_571c61e5a382457cbe5deececb435567 [id=a42dd4a1d931496f8786c21f55f1c87e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.221213 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.221374 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 66df0522aa2842f68aa824571fa61b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:22.757279 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a59d7fb5dc54b479d965282b1c7d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.221717 894 tablet_service.cc:1468] Processing CreateTablet for tablet 421b28d047124222ae9cbc603ec22cee (DEFAULT_TABLE table=default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.221869 887 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7086c5bfa4644d08e4f55231b85acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.878324 1547 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" instance_seqno: 1757101933934195) as {username='slave'} at 127.0.104.3:56757; Asking this server to re-register.
I20250905 19:52:23.222527 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 3 candidate_status { last_received { term: 2 index: 504 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:22.731590 1019 tablet_service.cc:1468] Processing CreateTablet for tablet e10b4e82c4bc429b9a42db39c03a5267 (DEFAULT_TABLE table=default.Table_1c3606c3e6e0452b910b8b3ce847fb58 [id=6a2d8d9aa80544c6b70397d8f061865b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.736589 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 91396422d0bb468395db04aa80262882 (DEFAULT_TABLE table=default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.223201 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 3 candidate_status { last_received { term: 2 index: 504 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:23.223289 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91396422d0bb468395db04aa80262882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:22.715179 1186 tablet_replica.cc:331] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:22.710922 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_cc71e9d45e1b4857a7f86f100e2d6376"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:22.757493 883 tablet_service.cc:1468] Processing CreateTablet for tablet fd1280fefc3e43db9928567a734644f1 (DEFAULT_TABLE table=default.Table_c8afd57df21440b5923aef4dd927ae45 [id=4447588b8d2443e5a561758689326e26]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.757812 879 tablet_service.cc:1468] Processing CreateTablet for tablet 56e7ab709fe04479a7b1f17dba2fd603 (DEFAULT_TABLE table=default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:22.763677 1706 log.cc:927] Time spent T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.076s user 0.000s sys 0.000s
I20250905 19:52:22.770666 1760 tablet.cc:1722] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:22.718168 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 66df0522aa2842f68aa824571fa61b05 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:22.711318 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_68450db54c5641a1a249cdcf5fa2d5ec" } modify_external_catalogs: true
I20250905 19:52:23.225551 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_68450db54c5641a1a249cdcf5fa2d5ec" } modify_external_catalogs: true
I20250905 19:52:23.225718 1171 tablet_bootstrap.cc:492] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:23.225836 1171 ts_tablet_manager.cc:1397] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 1.199s user 0.048s sys 0.075s
I20250905 19:52:23.226579 1171 raft_consensus.cc:357] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.226732 1171 raft_consensus.cc:383] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:23.226802 1171 raft_consensus.cc:738] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.226971 1171 consensus_queue.cc:260] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.227192 1098 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:23.227351 1171 ts_tablet_manager.cc:1428] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250905 19:52:22.711668 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_97c63bdbe93a48708207c1e9a96cb1bd" } modify_external_catalogs: true
I20250905 19:52:23.227629 1171 tablet_bootstrap.cc:492] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:22.718899 821 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:23.227653 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_97c63bdbe93a48708207c1e9a96cb1bd" } modify_external_catalogs: true
I20250905 19:52:22.815049 1184 ts_tablet_manager.cc:1918] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:22.711841 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f6e643730c10465cb8585cb48c1c5e04" } modify_external_catalogs: true
I20250905 19:52:23.228082 1184 log.cc:1199] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ffa14750f1b94c589779f7da0e764ccc
I20250905 19:52:23.228087 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f6e643730c10465cb8585cb48c1c5e04" } modify_external_catalogs: true
I20250905 19:52:23.228524 1184 ts_tablet_manager.cc:1939] T ffa14750f1b94c589779f7da0e764ccc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:22.718834 739 tablet_service.cc:1468] Processing CreateTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 (DEFAULT_TABLE table=default.Table_68450db54c5641a1a249cdcf5fa2d5ec [id=94f2531a27494c32bcd549a25287867f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.816325 1027 tablet_service.cc:1468] Processing CreateTablet for tablet d59da39e7bcf497a93fb9174328c1814 (DEFAULT_TABLE table=default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.724287 743 tablet_service.cc:1468] Processing CreateTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 (DEFAULT_TABLE table=default.Table_b21bb0be9ae1435c8929de3f6b662d8c [id=a9a1d004ba9f4415b2fb958c789d66ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.818310 1030 tablet_service.cc:1468] Processing CreateTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 (DEFAULT_TABLE table=default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.712963 448 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]). Expected version 2 got 0
I20250905 19:52:22.647306 1169 raft_consensus.cc:383] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:22.725922 744 tablet_service.cc:1468] Processing CreateTablet for tablet d495b6ba92e84b588b4780c2896195c0 (DEFAULT_TABLE table=default.Table_ad054f6da70449ecad529f69c6aa9705 [id=c31b7e937b6d4adfb299ae8e46330d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.727934 1170 tablet_bootstrap.cc:492] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:23.229249 1169 raft_consensus.cc:738] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.229380 1170 ts_tablet_manager.cc:1397] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 1.382s user 0.051s sys 0.116s
I20250905 19:52:22.772406 1769 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:23.229526 1169 consensus_queue.cc:260] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.773201 1586 raft_consensus.cc:491] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader f21c498a0d964e8fbebfd1b3cb1916fe)
I20250905 19:52:22.851732 1728 tablet.cc:1722] T af5748bea3bd4f51a0d6c28a1d9044a0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:23.229929 1586 raft_consensus.cc:513] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:22.778112 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 (DEFAULT_TABLE table=default.Table_68450db54c5641a1a249cdcf5fa2d5ec [id=94f2531a27494c32bcd549a25287867f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.230230 1170 raft_consensus.cc:357] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.230398 1170 raft_consensus.cc:383] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:22.780874 889 tablet_service.cc:1468] Processing CreateTablet for tablet 2568b9f6f37b48efbe92cef4c19d5dfb (DEFAULT_TABLE table=default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.230489 1170 raft_consensus.cc:738] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.230582 1586 leader_election.cc:290] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:22.781562 1768 tablet.cc:1722] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:22.786088 1185 ts_tablet_manager.cc:1918] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:23.230693 1170 consensus_queue.cc:260] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.230921 1185 log.cc:1199] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:22.855057 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 862214a31059407baefef8a390aaea69 (DEFAULT_TABLE table=default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.857477 1773 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:23.231288 1185 ts_tablet_manager.cc:1939] T 57f851a6eecc4a41806bdaa19cc5afab P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:23.231357 1186 raft_consensus.cc:2241] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:22.866283 1778 tablet.cc:1717] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 4eb3af79b01d4835bf6bf7b26bf016f5: Skipping requested alter to schema version 2, tablet already version 2
I20250905 19:52:23.231521 1186 raft_consensus.cc:2270] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:22.824487 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_78bb4fad681f4443a890340c78ec736d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:23.232136 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 304) took 2537 ms. Trace:
I20250905 19:52:23.232226 1185 rpcz_store.cc:276] 0905 19:52:20.695018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.695072 (+ 54us) service_pool.cc:225] Handling call
0905 19:52:22.085545 (+1390473us) ts_tablet_manager.cc:1145] Deleting tablet 57f851a6eecc4a41806bdaa19cc5afab
0905 19:52:22.085561 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:22.549076 (+463515us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:22.785937 (+236861us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.232082 (+446145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1326631}
W20250905 19:52:23.232409 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_78bb4fad681f4443a890340c78ec736d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.886114 1730 tablet.cc:1722] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:23.232721 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 308) took 2474 ms. Trace:
I20250905 19:52:22.945585 1022 tablet_service.cc:1468] Processing CreateTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 (DEFAULT_TABLE table=default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.232800 1185 rpcz_store.cc:276] 0905 19:52:20.758670 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.758829 (+ 159us) service_pool.cc:225] Handling call
0905 19:52:23.232544 (+2473715us) ts_tablet_manager.cc:1145] Deleting tablet 723168b075874dfcb3eb50c5134c499e
0905 19:52:23.232553 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.232672 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2473487}
I20250905 19:52:22.947793 1025 tablet_service.cc:1468] Processing CreateTablet for tablet af10abf0a5b34b3bbfea20e5c2005006 (DEFAULT_TABLE table=default.Table_7b30c5a5c660490191d00c27224d516f [id=8213b623d8414c62874496694b1515fd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:23.233083 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cc71e9d45e1b4857a7f86f100e2d6376 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.233160 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 310) took 2474 ms. Trace:
I20250905 19:52:23.233245 1185 rpcz_store.cc:276] 0905 19:52:20.758971 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.759100 (+ 129us) service_pool.cc:225] Handling call
0905 19:52:23.233016 (+2473916us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:23.233024 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.233112 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2473706}
I20250905 19:52:22.728169 746 tablet_service.cc:1468] Processing CreateTablet for tablet 91396422d0bb468395db04aa80262882 (DEFAULT_TABLE table=default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.233532 1185 tablet_replica.cc:331] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:22.825109 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_c14282695b114ab4b1d2e98d4d6d78fb"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:22.790966 893 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51212 (request call id 356) took 1070 ms. Trace:
I20250905 19:52:23.233706 893 rpcz_store.cc:276] 0905 19:52:21.720138 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.731800 (+ 11662us) service_pool.cc:225] Handling call
0905 19:52:21.737910 (+ 6110us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:21.737935 (+ 25us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:22.275807 (+537872us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:22.790742 (+514935us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:22.790908 (+ 166us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:52:23.233963 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c14282695b114ab4b1d2e98d4d6d78fb in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.234131 893 tablet_service.cc:1468] Processing CreateTablet for tablet be1315824fd749049a62af3e6a113b9a (DEFAULT_TABLE table=default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.793921 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.184782 1729 raft_consensus.cc:491] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.234972 1729 raft_consensus.cc:513] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.235157 884 tablet_service.cc:1468] Processing CreateTablet for tablet e5fc2c2ba49444af939f5a975165362b (DEFAULT_TABLE table=default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.797302 886 tablet_service.cc:1468] Processing CreateTablet for tablet 160a7814abf9423dbe965c252c20fa7f (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.811081 877 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.235638 1729 leader_election.cc:290] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:22.814545 1702 consensus_queue.cc:1035] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:23.235882 877 tablet_service.cc:1468] Processing CreateTablet for tablet e10b4e82c4bc429b9a42db39c03a5267 (DEFAULT_TABLE table=default.Table_1c3606c3e6e0452b910b8b3ce847fb58 [id=6a2d8d9aa80544c6b70397d8f061865b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.728048 751 tablet_service.cc:1468] Processing CreateTablet for tablet be1315824fd749049a62af3e6a113b9a (DEFAULT_TABLE table=default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.820297 1767 tablet.cc:1717] T 4fac0e1f483b40ab8d66cb5a9ee808fc P 835daf30e4324782848290be181ae78f: Skipping requested alter to schema version 2, tablet already version 2
W20250905 19:52:22.837327 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e44e88a0df444fe5b156a8fdf3199507 consensus state has no leader
I20250905 19:52:22.838117 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_ac8e97a442114459a1ff85613d8651f9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.236629 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ac8e97a442114459a1ff85613d8651f9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:22.830490 876 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.237303 876 tablet_service.cc:1468] Processing CreateTablet for tablet 91396422d0bb468395db04aa80262882 (DEFAULT_TABLE table=default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.728387 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a7814abf9423dbe965c252c20fa7f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:22.759037 440 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd1280fefc3e43db9928567a734644f1 because the tablet was not found. No further retry: Not found: Tablet not found: fd1280fefc3e43db9928567a734644f1
I20250905 19:52:23.147806 1685 consensus_queue.cc:1035] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:23.237578 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 478541us
I20250905 19:52:23.148046 1494 consensus_queue.cc:1035] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:23.237910 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 29c2b030fafd44ad9e648b2527a4ad07 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 29c2b030fafd44ad9e648b2527a4ad07 already in progress: creating tablet
W20250905 19:52:23.238510 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d296428b70f9411190b6c408d81639f2 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet d296428b70f9411190b6c408d81639f2 already in progress: creating tablet
W20250905 19:52:23.238623 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 723168b075874dfcb3eb50c5134c499e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 723168b075874dfcb3eb50c5134c499e already in progress: creating tablet
W20250905 19:52:23.238835 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ac5c88cf8d15433a940e937f0a225582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac5c88cf8d15433a940e937f0a225582 already in progress: creating tablet
I20250905 19:52:23.239491 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:23.239667 914 raft_consensus.cc:2466] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:22.729487 749 tablet_service.cc:1468] Processing CreateTablet for tablet ca646f7c162e41469f283657e72823b4 (DEFAULT_TABLE table=default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.731655 756 tablet_service.cc:1468] Processing CreateTablet for tablet e5fc2c2ba49444af939f5a975165362b (DEFAULT_TABLE table=default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:22.733508 1708 tablet.cc:1722] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:22.737072 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ae83eea5f74e9187713e8b82c9a23b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:22.745458 1772 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
W20250905 19:52:22.731513 712 leader_election.cc:343] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:23.240847 712 leader_election.cc:304] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:23.067212 698 kernel_stack_watchdog.cc:198] Thread 712 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 103ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb0da75e980 (unknown) at ??:0
@ 0x7fb0da7589c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fb0d4593904 google::LogMessage::Flush() at ??:0
@ 0x7fb0d4593a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb0e2d04973 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7fb0e2d074ea kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7fb0e2d072ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fb0de02ec4f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fb0de02e82e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fb0de01a860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fb0de01adac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fb0ddfb62aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fb0ddfb4c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fb0ddfc7135 ev::base<>::method_thunk<>() at ??:0
I20250905 19:52:23.242022 959 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:23.236604 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:23.242681 512 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:23.212872 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36f966bf512d4cbe9ae406386db31533" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:23.242998 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:23.244148 984 leader_election.cc:343] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.200366 428 kernel_stack_watchdog.cc:198] Thread 440 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 168ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f8bd732a980 (unknown) at ??:0
@ 0x7f8bd73249c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f8bd115f904 google::LogMessage::Flush() at ??:0
@ 0x7f8bd115fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f8be7568c1b kudu::master::AsyncDeleteReplica::HandleResponse() at ??:0
@ 0x7f8be74c2782 kudu::master::RetryingTSRpcTask::RpcCallback() at ??:0
@ 0x7f8be756c6e0 _ZZN4kudu6master18AsyncDeleteReplica11SendRequestEiENKUlvE_clEv at ??:0
@ 0x7f8be756c4fe _ZNSt17_Function_handlerIFvvEZN4kudu6master18AsyncDeleteReplica11SendRequestEiEUlvE_E9_M_invokeERKSt9_Any_data at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f8bdabfac4f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f8bdabfa82e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f8bdabe6860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f8bdabe6dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f8bdab822aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f8bdab80c66 kudu::rpc::Connection::ReadHandler() at ??:0
I20250905 19:52:23.244828 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:23.244326 984 leader_election.cc:304] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.245500 986 leader_election.cc:304] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:23.256245 984 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:23.247536 1186 ts_tablet_manager.cc:1905] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.256724 1729 raft_consensus.cc:2747] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:23.252200 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e7086c5bfa4644d08e4f55231b85acf4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.256912 1729 raft_consensus.cc:2802] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:23.257005 1729 raft_consensus.cc:491] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:23.257086 1729 raft_consensus.cc:3058] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:23.258389 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36f966bf512d4cbe9ae406386db31533" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:23.250803 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421b28d047124222ae9cbc603ec22cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.260454 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d7155cf5abba4e2cbdce583406f94fad. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.263744 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.264554 1547 ts_manager.cc:194] Registered new tserver with Master: f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:23.264902 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2568b9f6f37b48efbe92cef4c19d5dfb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.265482 879 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 56e7ab709fe04479a7b1f17dba2fd603. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.266044 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ebab2e3698f4f60a770bde9cc019600. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:23.266355 848 leader_election.cc:343] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.266754 848 consensus_peers.cc:489] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:23.267177 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae6fe4620dda4b1296416e01f8ffee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.268108 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 160a7814abf9423dbe965c252c20fa7f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.268486 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10b4e82c4bc429b9a42db39c03a5267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.268790 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd1280fefc3e43db9928567a734644f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.269815 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91396422d0bb468395db04aa80262882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.271689 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1315824fd749049a62af3e6a113b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.275375 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10b4e82c4bc429b9a42db39c03a5267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.274315 1729 raft_consensus.cc:513] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.276476 884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5fc2c2ba49444af939f5a975165362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.274806 1769 tablet.cc:1722] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:23.278162 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fa5d34f9c0ee4d55ade61b30b0f4c293. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.279973 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5302b79208494e48a5f2778c5f52ad01. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.281849 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:23.282079 913 raft_consensus.cc:3058] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:23.283450 1729 leader_election.cc:290] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:23.285712 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59da39e7bcf497a93fb9174328c1814. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:23.291198 441 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 03a12cf719f34096bb82f5846bbe3b2d (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03a12cf719f34096bb82f5846bbe3b2d consensus state has no leader
I20250905 19:52:23.293815 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "36f966bf512d4cbe9ae406386db31533" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:23.294000 1045 raft_consensus.cc:3058] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:23.296187 849 leader_election.cc:343] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: ffa14750f1b94c589779f7da0e764ccc
I20250905 19:52:23.302352 882 tablet_service.cc:1468] Processing CreateTablet for tablet ca646f7c162e41469f283657e72823b4 (DEFAULT_TABLE table=default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.302767 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca646f7c162e41469f283657e72823b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.306876 448 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_bfc0123a4cd9449ba4bcb59bf2e88ddb [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 4 got 3
I20250905 19:52:23.307124 448 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]). Expected version 2 got 0
I20250905 19:52:23.308722 1045 raft_consensus.cc:2466] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:23.308872 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ac13b19fb6427a9ef35a5160033a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.309082 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862214a31059407baefef8a390aaea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.309757 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 309) took 2566 ms. Trace:
I20250905 19:52:23.309890 1184 rpcz_store.cc:276] 0905 19:52:20.743247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.743300 (+ 53us) service_pool.cc:225] Handling call
0905 19:52:22.653240 (+1909940us) ts_tablet_manager.cc:1145] Deleting tablet ffa14750f1b94c589779f7da0e764ccc
0905 19:52:22.653250 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:22.753741 (+100491us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:22.813899 (+ 60158us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.309698 (+495799us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1900990}
I20250905 19:52:23.310437 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 310) took 2567 ms. Trace:
I20250905 19:52:23.310560 1184 rpcz_store.cc:276] 0905 19:52:20.743349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.743410 (+ 61us) service_pool.cc:225] Handling call
0905 19:52:23.310225 (+2566815us) ts_tablet_manager.cc:1145] Deleting tablet 723168b075874dfcb3eb50c5134c499e
0905 19:52:23.310234 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.310383 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2557877}
I20250905 19:52:23.310871 1184 tablet_replica.cc:331] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:23.311115 1184 raft_consensus.cc:2241] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:23.311300 1184 raft_consensus.cc:2270] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.311755 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e10b4e82c4bc429b9a42db39c03a5267. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.312320 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 3609de83732d4ad8857372acfbb5c32a
I20250905 19:52:23.315570 1547 ts_manager.cc:194] Registered new tserver with Master: 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:23.316287 449 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6ebab2e3698f4f60a770bde9cc019600
I20250905 19:52:23.316392 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af10abf0a5b34b3bbfea20e5c2005006. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.317890 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 0d4204f61929498188c18529148a8d13
I20250905 19:52:23.320463 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a11777cf2d4ea9a5cdf792f73aff76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.328982 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ffa14750f1b94c589779f7da0e764ccc (table default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685]) successfully deleted
W20250905 19:52:23.329281 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 723168b075874dfcb3eb50c5134c499e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 723168b075874dfcb3eb50c5134c499e already in progress: creating tablet
I20250905 19:52:23.330699 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4204f61929498188c18529148a8d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:23.330693 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:23.332150 1184 ts_tablet_manager.cc:1905] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:23.333424 848 leader_election.cc:343] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:23.344779 849 leader_election.cc:304] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.346563 1185 raft_consensus.cc:2241] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:23.347183 1703 raft_consensus.cc:2802] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:23.352988 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:23.357698 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a73510a5e598492c83e3a2d2ee35d95d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.358839 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a73510a5e598492c83e3a2d2ee35d95d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.359356 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4797854b3d9244bd8aac427a5de6985a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.359848 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4797854b3d9244bd8aac427a5de6985a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.362743 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:23.363224 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 421b28d047124222ae9cbc603ec22cee. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:23.363292 984 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:23.364612 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca646f7c162e41469f283657e72823b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.365787 449 catalog_manager.cc:5582] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:23.366155 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]). Expected version 1 got 0
I20250905 19:52:23.366369 449 catalog_manager.cc:5582] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:23.366842 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 1 got 0
I20250905 19:52:23.367039 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_bfc0123a4cd9449ba4bcb59bf2e88ddb [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 4 got 3
I20250905 19:52:23.367266 449 catalog_manager.cc:5582] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:23.367622 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac]). Expected version 2 got 0
I20250905 19:52:23.367900 449 catalog_manager.cc:5582] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:23.368288 449 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:23.368994 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5fc2c2ba49444af939f5a975165362b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.370302 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d5eb8bec074d2c8766fe7b159031e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.370623 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 91396422d0bb468395db04aa80262882. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.373811 1703 raft_consensus.cc:695] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:23.376993 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d495b6ba92e84b588b4780c2896195c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.374130 1703 consensus_queue.cc:237] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.378207 1760 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51212 (request call id 340) took 1946 ms. Trace:
I20250905 19:52:23.379707 890 tablet_service.cc:1468] Processing CreateTablet for tablet d59da39e7bcf497a93fb9174328c1814 (DEFAULT_TABLE table=default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.382143 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ebab2e3698f4f60a770bde9cc019600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.382794 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet be1315824fd749049a62af3e6a113b9a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.384814 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a59d7fb5dc54b479d965282b1c7d24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.385330 913 raft_consensus.cc:2466] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:23.390008 986 leader_election.cc:304] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.390352 1798 raft_consensus.cc:2802] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:23.390512 1798 raft_consensus.cc:695] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:23.390784 1798 consensus_queue.cc:237] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.391386 1184 ts_tablet_manager.cc:1918] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:23.391518 1184 log.cc:1199] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:23.392102 1768 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51212 (request call id 363) took 1421 ms. Trace:
I20250905 19:52:23.400022 894 tablet_service.cc:1468] Processing CreateTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 (DEFAULT_TABLE table=default.Table_b21bb0be9ae1435c8929de3f6b662d8c [id=a9a1d004ba9f4415b2fb958c789d66ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.402834 1547 ts_manager.cc:194] Registered new tserver with Master: 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:23.391939 1184 ts_tablet_manager.cc:1939] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:23.379243 1185 raft_consensus.cc:2270] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.412111 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 311) took 2668 ms. Trace:
I20250905 19:52:23.412227 1184 rpcz_store.cc:276] 0905 19:52:20.743762 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.743816 (+ 54us) service_pool.cc:225] Handling call
0905 19:52:23.310804 (+2566988us) ts_tablet_manager.cc:1145] Deleting tablet 19e4edddafcd4d22a0e0ae66b0b953e0
0905 19:52:23.310815 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.335953 (+ 25138us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.391283 (+ 55330us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.412051 (+ 20768us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2558393}
I20250905 19:52:23.412359 1185 ts_tablet_manager.cc:1905] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.412568 1184 tablet_replica.cc:331] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:23.412659 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 19e4edddafcd4d22a0e0ae66b0b953e0 (table default.Table_ff5415e10957422498084d7730365989 [id=bd0b3b55bf3648b7a45b7aef904be570]) successfully deleted
I20250905 19:52:23.412755 1184 raft_consensus.cc:2241] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:23.412923 1184 raft_consensus.cc:2270] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.413448 1184 ts_tablet_manager.cc:1905] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.413901 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_571c61e5a382457cbe5deececb435567" } new_table_name: "default.Table_2e5e3cf0f8994adfb2837e2fa148fd98" modify_external_catalogs: true
I20250905 19:52:23.415306 451 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f542b700d5364bfeb6e1e0111834fab0" } modify_external_catalogs: true
I20250905 19:52:23.415477 451 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_f542b700d5364bfeb6e1e0111834fab0" } modify_external_catalogs: true
I20250905 19:52:23.416544 1169 ts_tablet_manager.cc:1428] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.771s user 0.007s sys 0.007s
W20250905 19:52:23.416742 1169 ts_tablet_manager.cc:1460] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Tablet startup took 771ms
I20250905 19:52:23.416841 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fe03bfebc206445e842984b466de6d91" } modify_external_catalogs: true
I20250905 19:52:23.416982 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_fe03bfebc206445e842984b466de6d91" } modify_external_catalogs: true
W20250905 19:52:23.416932 1169 ts_tablet_manager.cc:1462] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Trace:
0905 19:52:17.836815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:17.837466 (+ 651us) service_pool.cc:225] Handling call
0905 19:52:17.839519 (+ 2053us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:17.839544 (+ 25us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:17.875078 (+ 35534us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:17.878052 (+ 2974us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:17.878249 (+ 197us) inbound_call.cc:173] Queueing success response
0905 19:52:21.613173 (+3734924us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0905 19:52:22.645146 (+1031973us) ts_tablet_manager.cc:1429] Starting tablet replica
0905 19:52:22.645635 (+ 489us) tablet_replica.cc:260] Starting instrumentation
0905 19:52:22.645995 (+ 360us) tablet_replica.cc:279] Starting consensus
Metrics: {"raft.queue_time_us":6290,"raft.run_cpu_time_us":52,"raft.run_wall_time_us":52,"tablet-open.queue_time_us":3734743,"thread_start_us":175925,"threads_started":1}
I20250905 19:52:23.417949 1169 tablet_bootstrap.cc:492] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:23.348242 876 tablet_service.cc:1468] Processing CreateTablet for tablet d495b6ba92e84b588b4780c2896195c0 (DEFAULT_TABLE table=default.Table_ad054f6da70449ecad529f69c6aa9705 [id=c31b7e937b6d4adfb299ae8e46330d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.422087 453 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_bfc0123a4cd9449ba4bcb59bf2e88ddb" } new_table_name: "default.Table_cb111dc4aa3946e8a89529d1cd623c08" modify_external_catalogs: true
I20250905 19:52:23.423223 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1c3606c3e6e0452b910b8b3ce847fb58" } new_table_name: "default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9" modify_external_catalogs: true
W20250905 19:52:23.423247 849 consensus_peers.cc:489] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4c76077e4adf45a7adc736b0e69032eb. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:23.425316 848 leader_election.cc:343] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: ffa14750f1b94c589779f7da0e764ccc
I20250905 19:52:23.429247 1171 tablet_bootstrap.cc:654] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:23.429718 448 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31] alter complete (version 2)
I20250905 19:52:23.430379 848 leader_election.cc:304] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.433367 880 tablet_service.cc:1468] Processing CreateTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 (DEFAULT_TABLE table=default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.435163 455 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d] alter complete (version 2)
I20250905 19:52:23.434583 875 tablet_service.cc:1468] Processing CreateTablet for tablet 862214a31059407baefef8a390aaea69 (DEFAULT_TABLE table=default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.438304 892 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51212 (request call id 375) took 1092 ms. Trace:
I20250905 19:52:23.438402 892 rpcz_store.cc:276] 0905 19:52:22.324156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.324707 (+ 551us) service_pool.cc:225] Handling call
0905 19:52:22.593698 (+268991us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:22.593726 (+ 28us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.407655 (+813929us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.416545 (+ 8890us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.416715 (+ 170us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:23.439445 892 tablet_service.cc:1468] Processing CreateTablet for tablet af10abf0a5b34b3bbfea20e5c2005006 (DEFAULT_TABLE table=default.Table_7b30c5a5c660490191d00c27224d516f [id=8213b623d8414c62874496694b1515fd]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:23.439793 848 raft_consensus.cc:2684] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250905 19:52:23.440429 848 consensus_peers.cc:489] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:23.440626 848 consensus_peers.cc:489] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:23.441416 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59da39e7bcf497a93fb9174328c1814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.441402 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.441641 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.441838 879 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4204f61929498188c18529148a8d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.442057 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862214a31059407baefef8a390aaea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.430976 1768 rpcz_store.cc:276] 0905 19:52:21.970702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.996060 (+ 25358us) service_pool.cc:225] Handling call
0905 19:52:23.392063 (+1396003us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:21.996620 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:21.996718 (+ 98us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:21.996727 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:21.996734 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:21.996771 (+ 37us) alter_schema_op.cc:128] START. Timestamp: 1757101941996747
0905 19:52:21.996773 (+ 2us) op_driver.cc:348] REPLICATION: starting
0905 19:52:21.996955 (+ 182us) log.cc:844] Serialized 173 byte log entry
0905 19:52:22.696324 (+699369us) op_driver.cc:464] REPLICATION: finished
0905 19:52:22.781504 (+ 85180us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:23.269241 (+487737us) log.cc:844] Serialized 5 byte log entry
0905 19:52:23.389264 (+120023us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.389543 (+ 279us) log.cc:844] Serialized 14 byte log entry
0905 19:52:23.392024 (+ 2481us) alter_schema_op.cc:73] Released schema lock
0905 19:52:23.392030 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":85078,"prepare.queue_time_us":104,"prepare.run_cpu_time_us":463,"prepare.run_wall_time_us":910,"raft.queue_time_us":116,"raft.run_cpu_time_us":364,"raft.run_wall_time_us":354,"replication_time_us":699512,"thread_start_us":86365,"threads_started":1}]]}
I20250905 19:52:23.447063 1809 raft_consensus.cc:491] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.447222 1809 raft_consensus.cc:513] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.448521 1809 leader_election.cc:290] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:23.449625 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 14d5eb8bec074d2c8766fe7b159031e5. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.449954 1184 ts_tablet_manager.cc:1918] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:23.450075 1184 log.cc:1199] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:23.450510 1184 ts_tablet_manager.cc:1939] T a5f076d9917f40d899d7422ad974fe0f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:23.450587 881 tablet_service.cc:1468] Processing CreateTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 (DEFAULT_TABLE table=default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.451242 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 308) took 2708 ms. Trace:
I20250905 19:52:23.451352 1184 rpcz_store.cc:276] 0905 19:52:20.742954 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.752036 (+ 9082us) service_pool.cc:225] Handling call
0905 19:52:23.412483 (+2660447us) ts_tablet_manager.cc:1145] Deleting tablet a5f076d9917f40d899d7422ad974fe0f
0905 19:52:23.412493 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.429735 (+ 17242us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.449818 (+ 20083us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.451203 (+ 1385us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2653888}
I20250905 19:52:23.451782 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a5f076d9917f40d899d7422ad974fe0f (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:23.458509 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d495b6ba92e84b588b4780c2896195c0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.458441 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a98611b77e2d4e7c868fe9641069ae23"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.459012 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a98611b77e2d4e7c868fe9641069ae23 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.438058 1760 rpcz_store.cc:276] 0905 19:52:21.403813 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.403867 (+ 54us) service_pool.cc:225] Handling call
0905 19:52:23.350301 (+1946434us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:21.404690 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:21.404792 (+ 102us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:21.404801 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:21.404807 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:21.404841 (+ 34us) alter_schema_op.cc:128] START. Timestamp: 1757101941404816
0905 19:52:21.404843 (+ 2us) op_driver.cc:348] REPLICATION: starting
0905 19:52:21.404990 (+ 147us) log.cc:844] Serialized 173 byte log entry
0905 19:52:22.396288 (+991298us) op_driver.cc:464] REPLICATION: finished
0905 19:52:22.770587 (+374299us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:23.264166 (+493579us) log.cc:844] Serialized 5 byte log entry
0905 19:52:23.303348 (+ 39182us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.303626 (+ 278us) log.cc:844] Serialized 14 byte log entry
0905 19:52:23.350252 (+ 46626us) alter_schema_op.cc:73] Released schema lock
0905 19:52:23.350258 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":374200,"mutex_wait_us":3751,"prepare.queue_time_us":366,"prepare.run_cpu_time_us":424,"prepare.run_wall_time_us":4148,"raft.queue_time_us":3846,"raft.run_cpu_time_us":345,"raft.run_wall_time_us":344,"replication_time_us":991407,"spinlock_wait_cycles":41472,"thread_start_us":356737,"threads_started":1}]]}
I20250905 19:52:23.467156 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:23.467612 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ac13b19fb6427a9ef35a5160033a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.471431 1169 tablet_bootstrap.cc:654] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:23.472245 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ae6fe4620dda4b1296416e01f8ffee29 consensus state has no leader
I20250905 19:52:23.475296 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af10abf0a5b34b3bbfea20e5c2005006. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.476126 446 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_6a7e530e3936440fa898546212620a3b" } new_table_name: "default.Table_c2a680ef01e34dceb09eb62c5948d981" modify_external_catalogs: true
I20250905 19:52:23.451625 1767 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:51212 (request call id 334) took 2272 ms. Trace:
I20250905 19:52:23.477451 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_650ae210fee64c839acb6988b887b7c2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.478432 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_650ae210fee64c839acb6988b887b7c2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.478734 1767 rpcz_store.cc:276] 0905 19:52:21.172164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.172322 (+ 158us) service_pool.cc:225] Handling call
0905 19:52:23.444975 (+2272653us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:21.188077 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:21.188183 (+ 106us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:21.807128 (+618945us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:21.807141 (+ 13us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:21.807195 (+ 54us) alter_schema_op.cc:128] START. Timestamp: 1757101941807167
0905 19:52:21.807200 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:21.807439 (+ 239us) log.cc:844] Serialized 173 byte log entry
0905 19:52:22.701813 (+894374us) op_driver.cc:464] REPLICATION: finished
0905 19:52:22.820190 (+118377us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:23.236886 (+416696us) log.cc:844] Serialized 94 byte log entry
0905 19:52:23.444928 (+208042us) alter_schema_op.cc:73] Released schema lock
0905 19:52:23.444933 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":118304,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":32184,"prepare.run_wall_time_us":619870,"raft.queue_time_us":236,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":894583,"thread_start_us":287911,"threads_started":2,"wal-append.queue_time_us":223782}]]}
I20250905 19:52:23.479660 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a11777cf2d4ea9a5cdf792f73aff76. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:23.481205 986 leader_election.cc:343] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:23.481983 878 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51212 (request call id 383) took 1027 ms. Trace:
I20250905 19:52:23.492515 451 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet c46ea498d6a14d1ebdffea415cfc7114
I20250905 19:52:23.493182 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32fa16972f2a42d39b338b833c185b28. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.492043 878 rpcz_store.cc:276] 0905 19:52:22.441356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.441655 (+ 299us) service_pool.cc:225] Handling call
0905 19:52:23.212179 (+770524us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:23.212203 (+ 24us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.382605 (+170402us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.468338 (+ 85733us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.468482 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:23.502138 451 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_55ee6ff3b1904fd98253e035e91cb732" } new_table_name: "default.Table_e892ac4352f44019ab940683425afff7" modify_external_catalogs: true
I20250905 19:52:23.511435 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:23.514492 450 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:23.516552 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:23.516728 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 312) took 2714 ms. Trace:
I20250905 19:52:23.517369 878 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.519019 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fdc9d9815864aa7bf5ca9a5c4485390. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:23.521425 848 consensus_peers.cc:489] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:23.521728 1184 rpcz_store.cc:276] 0905 19:52:20.766753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.766802 (+ 49us) service_pool.cc:225] Handling call
0905 19:52:23.460061 (+2693259us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:23.460070 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.481578 (+ 21508us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2693060}
I20250905 19:52:23.522189 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 313) took 2697 ms. Trace:
I20250905 19:52:23.524494 1184 rpcz_store.cc:276] 0905 19:52:20.825098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.844774 (+ 19676us) service_pool.cc:225] Handling call
0905 19:52:23.522014 (+2677240us) ts_tablet_manager.cc:1145] Deleting tablet fd793bce71e344f18bb177a53ab1bfc1
0905 19:52:23.522025 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.522155 (+ 130us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2676950}
I20250905 19:52:23.524825 1184 tablet_replica.cc:331] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:23.527288 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.527524 877 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.527731 1184 raft_consensus.cc:2241] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:23.527925 1184 raft_consensus.cc:2270] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.528046 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:23.528468 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ac5c88cf8d15433a940e937f0a225582 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet ac5c88cf8d15433a940e937f0a225582 already in progress: creating tablet
I20250905 19:52:23.548076 449 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_cb111dc4aa3946e8a89529d1cd623c08 [id=5ac15645cb4d4730a4b4464a85299bed]). Expected version 5 got 4
W20250905 19:52:23.864171 441 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 335679us
W20250905 19:52:23.864493 441 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet fd793bce71e344f18bb177a53ab1bfc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd793bce71e344f18bb177a53ab1bfc1 already in progress: creating tablet
I20250905 19:52:23.865554 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_e32bc15eee6447be8dcf60f46cc58e12"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:23.866330 455 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b21bb0be9ae1435c8929de3f6b662d8c" } modify_external_catalogs: true
I20250905 19:52:23.866523 455 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_b21bb0be9ae1435c8929de3f6b662d8c" } modify_external_catalogs: true
I20250905 19:52:23.564321 1783 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20250905 19:52:23.867050 448 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ee3eaf9eb84d435187c73a394308861b" } modify_external_catalogs: true
I20250905 19:52:23.542331 735 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:48700 (request call id 91) took 1216 ms. Trace:
I20250905 19:52:23.867189 448 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ee3eaf9eb84d435187c73a394308861b" } modify_external_catalogs: true
I20250905 19:52:23.867237 735 rpcz_store.cc:276] 0905 19:52:22.294836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.294900 (+ 64us) service_pool.cc:225] Handling call
0905 19:52:22.680583 (+385683us) spinlock_profiling.cc:243] Waited 378 ms on lock 0x616000122854. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:22.702591 (+ 22008us) spinlock_profiling.cc:243] Waited 15.1 ms on lock 0x6160001a1754. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:23.511058 (+808467us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":904116864}
I20250905 19:52:23.528311 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:23.536769 1184 ts_tablet_manager.cc:1905] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.867707 453 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3fa477c60f534a63b2cd4e54fe7c5c40" } modify_external_catalogs: true
I20250905 19:52:23.867882 453 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_3fa477c60f534a63b2cd4e54fe7c5c40" } modify_external_catalogs: true
I20250905 19:52:23.868384 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_b543aba1b13749578be5d9e1dd312c0d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.868911 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b543aba1b13749578be5d9e1dd312c0d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.546146 1170 ts_tablet_manager.cc:1428] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.283s user 0.017s sys 0.009s
W20250905 19:52:23.870844 1170 ts_tablet_manager.cc:1460] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Tablet startup took 641ms
W20250905 19:52:23.870985 1170 ts_tablet_manager.cc:1462] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Trace:
0905 19:52:16.501009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:16.504025 (+ 3016us) service_pool.cc:225] Handling call
0905 19:52:16.542210 (+ 38185us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:16.542230 (+ 20us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:16.578691 (+ 36461us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:16.637919 (+ 59228us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:16.658296 (+ 20377us) inbound_call.cc:173] Queueing success response
0905 19:52:21.847581 (+5189285us) ts_tablet_manager.cc:1373] Bootstrapping tablet
0905 19:52:23.229461 (+1381880us) ts_tablet_manager.cc:1429] Starting tablet replica
0905 19:52:23.229686 (+ 225us) tablet_replica.cc:260] Starting instrumentation
0905 19:52:23.229808 (+ 122us) tablet_replica.cc:279] Starting consensus
Metrics: {"mutex_wait_us":20208,"raft.queue_time_us":295580,"raft.run_cpu_time_us":231,"raft.run_wall_time_us":5105,"tablet-open.queue_time_us":5209532,"thread_start_us":250889,"threads_started":1}
I20250905 19:52:23.871641 1170 tablet_bootstrap.cc:492] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:23.542366 1584 raft_consensus.cc:491] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.872217 1584 raft_consensus.cc:513] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.872939 1584 leader_election.cc:290] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:23.539369 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:23.556341 736 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:48700 (request call id 87) took 1857 ms. Trace:
I20250905 19:52:23.876881 736 rpcz_store.cc:276] 0905 19:52:21.665554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.665660 (+ 106us) service_pool.cc:225] Handling call
0905 19:52:22.677133 (+1011473us) spinlock_profiling.cc:243] Waited 980 ms on lock 0x616000122854. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:22.701960 (+ 24827us) spinlock_profiling.cc:243] Waited 15.6 ms on lock 0x6160001a1754. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:23.523439 (+821479us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2290645504}
I20250905 19:52:23.583036 1767 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
W20250905 19:52:23.879177 836 kernel_stack_watchdog.cc:198] Thread 848 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 220ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f18ad6fa980 (unknown) at ??:0
@ 0x7f18ad6f49c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f18a752f904 google::LogMessage::Flush() at ??:0
@ 0x7f18a752fa11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f18b5c37c6e kudu::consensus::Peer::ProcessResponseErrorUnlocked() at ??:0
@ 0x7f18b5c36d61 kudu::consensus::Peer::ProcessResponse() at ??:0
@ 0x7f18b5c3d521 kudu::consensus::Peer::SendNextRequest()::$_3::operator()() at ??:0
@ 0x7f18b5c3d32e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f18b0fcac4f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f18b0fca82e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f18b0fb6860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f18b0fb6dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f18b0f522aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f18b0f50c66 kudu::rpc::Connection::ReadHandler() at ??:0
W20250905 19:52:23.592715 848 consensus_peers.cc:489] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:23.880923 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 293454us
I20250905 19:52:23.577702 1169 tablet_bootstrap.cc:492] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:23.584964 1728 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:23.881826 1169 ts_tablet_manager.cc:1397] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.464s user 0.006s sys 0.006s
I20250905 19:52:23.619621 1171 tablet_bootstrap.cc:492] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:23.882037 1171 ts_tablet_manager.cc:1397] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.654s user 0.010s sys 0.006s
I20250905 19:52:23.679275 1799 raft_consensus.cc:491] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.882220 1799 raft_consensus.cc:513] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:23.882483 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e32bc15eee6447be8dcf60f46cc58e12 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.882737 1169 raft_consensus.cc:357] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.882787 1171 raft_consensus.cc:357] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.882944 1169 raft_consensus.cc:383] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:23.882983 1171 raft_consensus.cc:383] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:23.883023 1169 raft_consensus.cc:738] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.883064 1171 raft_consensus.cc:738] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.883257 1169 consensus_queue.cc:260] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.883296 1171 consensus_queue.cc:260] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.883493 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:23.883728 1169 ts_tablet_manager.cc:1428] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:23.883770 1171 ts_tablet_manager.cc:1428] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250905 19:52:23.559522 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 789771us
I20250905 19:52:23.884076 1171 tablet_bootstrap.cc:492] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:23.884716 1704 raft_consensus.cc:491] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.606949 1185 ts_tablet_manager.cc:1918] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:23.625248 888 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51212 (request call id 381) took 1196 ms. Trace:
I20250905 19:52:23.884881 1704 raft_consensus.cc:513] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.885038 1185 log.cc:1199] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ffa14750f1b94c589779f7da0e764ccc
I20250905 19:52:23.885538 1704 leader_election.cc:290] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:23.885600 1185 ts_tablet_manager.cc:1939] T ffa14750f1b94c589779f7da0e764ccc P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:23.620486 885 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:51212 (request call id 384) took 1086 ms. Trace:
I20250905 19:52:23.885912 885 rpcz_store.cc:276] 0905 19:52:22.533862 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.592327 (+ 58465us) service_pool.cc:225] Handling call
0905 19:52:22.660984 (+ 68657us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:22.661005 (+ 21us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.596445 (+935440us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.620306 (+ 23861us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.620450 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:23.886385 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 307) took 3127 ms. Trace:
I20250905 19:52:23.886515 1185 rpcz_store.cc:276] 0905 19:52:20.758508 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.759032 (+ 524us) service_pool.cc:225] Handling call
0905 19:52:23.233461 (+2474429us) ts_tablet_manager.cc:1145] Deleting tablet ffa14750f1b94c589779f7da0e764ccc
0905 19:52:23.233469 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.482623 (+249154us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.602280 (+119657us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.886341 (+284061us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":73070,"tablet-delete.queue_time_us":2474115}
I20250905 19:52:23.886947 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 306) took 3128 ms. Trace:
I20250905 19:52:23.887055 1185 rpcz_store.cc:276] 0905 19:52:20.758347 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.759372 (+ 1025us) service_pool.cc:225] Handling call
0905 19:52:23.886799 (+3127427us) ts_tablet_manager.cc:1145] Deleting tablet a5f076d9917f40d899d7422ad974fe0f
0905 19:52:23.886808 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.886909 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3127100}
I20250905 19:52:23.887198 1171 tablet_bootstrap.cc:654] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:23.887362 1185 tablet_replica.cc:331] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:23.887595 1185 raft_consensus.cc:2241] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:23.887805 1185 raft_consensus.cc:2270] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.885005 888 rpcz_store.cc:276] 0905 19:52:22.441086 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.448155 (+ 7069us) service_pool.cc:225] Handling call
0905 19:52:22.599393 (+151238us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:22.599421 (+ 28us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.617862 (+1018441us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.625088 (+ 7226us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.625214 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:52:23.889307 986 leader_election.cc:343] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.529359 984 leader_election.cc:343] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:23.891290 984 leader_election.cc:304] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.891585 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0edac849e8834894be2d4ff323edd13d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:23.891667 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 372711us
I20250905 19:52:23.562140 734 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:48700 (request call id 89) took 1831 ms. Trace:
I20250905 19:52:23.892082 1809 raft_consensus.cc:2747] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:23.892062 734 rpcz_store.cc:276] 0905 19:52:21.721102 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.721165 (+ 63us) service_pool.cc:225] Handling call
0905 19:52:22.677561 (+956396us) spinlock_profiling.cc:243] Waited 951 ms on lock 0x616000122854. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:22.702788 (+ 25227us) spinlock_profiling.cc:243] Waited 13.2 ms on lock 0x6160001a1754. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:23.552205 (+849417us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2216808960}
I20250905 19:52:23.606387 745 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48712 (request call id 379) took 1073 ms. Trace:
I20250905 19:52:23.892436 745 rpcz_store.cc:276] 0905 19:52:22.529049 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.585611 (+ 56562us) service_pool.cc:225] Handling call
0905 19:52:22.696466 (+110855us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:22.696511 (+ 45us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.554291 (+857780us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.598479 (+ 44188us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.602015 (+ 3536us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:23.619820 1186 ts_tablet_manager.cc:1918] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:23.892724 1186 log.cc:1199] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/cf2881d228964cd3bef040a06045d8d2
I20250905 19:52:23.893143 1186 ts_tablet_manager.cc:1939] T cf2881d228964cd3bef040a06045d8d2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:23.893473 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48712 (request call id 386) took 1283 ms. Trace:
I20250905 19:52:23.893554 1186 rpcz_store.cc:276] 0905 19:52:22.609888 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.633503 (+ 23615us) service_pool.cc:225] Handling call
0905 19:52:22.711174 (+ 77671us) ts_tablet_manager.cc:1145] Deleting tablet cf2881d228964cd3bef040a06045d8d2
0905 19:52:22.711185 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.597122 (+885937us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.606958 (+ 9836us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.893421 (+286463us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7585,"tablet-delete.queue_time_us":48}
I20250905 19:52:23.893886 1186 tablet_replica.cc:331] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:23.894068 1186 raft_consensus.cc:2241] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:23.894279 1186 raft_consensus.cc:2270] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.894891 1186 ts_tablet_manager.cc:1905] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.894923 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d480531ba94f4b0f9fe40e88fe91512e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:23.843361 970 kernel_stack_watchdog.cc:198] Thread 984 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 269ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f73f5b49980 (unknown) at ??:0
@ 0x7f73f5b439c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f73ef97e904 google::LogMessage::Flush() at ??:0
@ 0x7f73ef97ea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f73fe0ef973 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f73fe0f24ea kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f73fe0f22ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f73f9419c4f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f73f941982e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f73f9405860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f73f9405dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f73f93a12aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f73f939fc66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f73f93b2135 ev::base<>::method_thunk<>() at ??:0
I20250905 19:52:23.625963 733 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:48700 (request call id 90) took 1727 ms. Trace:
I20250905 19:52:23.896041 733 rpcz_store.cc:276] 0905 19:52:21.802409 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.802469 (+ 60us) service_pool.cc:225] Handling call
0905 19:52:22.677359 (+874890us) spinlock_profiling.cc:243] Waited 867 ms on lock 0x616000122854. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:22.702092 (+ 24733us) spinlock_profiling.cc:243] Waited 14.6 ms on lock 0x6160001a1754. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:23.381474 (+679382us) spinlock_profiling.cc:243] Waited 98 ms on lock 0x616000250d30. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e40b6859 00007fb0e6b02528 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:23.529556 (+148082us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2252703232}
I20250905 19:52:23.626008 1800 raft_consensus.cc:2747] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:23.896549 1800 raft_consensus.cc:491] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.896720 1800 raft_consensus.cc:513] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.897514 1800 leader_election.cc:290] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:23.626070 1708 tablet_replica.cc:1408] Time spent flushing the Log queue took a long time: real 0.288s user 0.008s sys 0.003s
I20250905 19:52:23.899421 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ffa14750f1b94c589779f7da0e764ccc (table default.Table_44155b42e0d04fc0b3a53308c495f845 [id=8a1b8eddc57e400696283fed538ae685]) successfully deleted
I20250905 19:52:23.626189 732 rpcz_store.cc:275] Call kudu.tserver.TabletServerService.ListTablets from 127.0.0.1:48700 (request call id 88) took 1862 ms. Trace:
I20250905 19:52:23.900048 732 rpcz_store.cc:276] 0905 19:52:21.687818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.687892 (+ 74us) service_pool.cc:225] Handling call
0905 19:52:22.677975 (+990083us) spinlock_profiling.cc:243] Waited 968 ms on lock 0x616000122854. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:22.702266 (+ 24291us) spinlock_profiling.cc:243] Waited 17.5 ms on lock 0x6160001a1754. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d68700 00007fb0e6b0271f 00007fb0e1233c36 00007fb0e1233965 00007fb0de0c00be 00007fb0de0bef9e 00007fb0de0c51d5 00007fb0de0c5aff 00007fb0de0c5928 0000000000a774dc
0905 19:52:23.550174 (+847908us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":2267766912}
I20250905 19:52:23.628008 739 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48712 (request call id 381) took 1026 ms. Trace:
I20250905 19:52:23.628008 750 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.CreateTablet from 127.0.0.1:48712 (request call id 375) took 1053 ms. Trace:
I20250905 19:52:23.901679 750 rpcz_store.cc:276] 0905 19:52:22.524149 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.525436 (+ 1287us) service_pool.cc:225] Handling call
0905 19:52:22.705773 (+180337us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:22.705846 (+ 73us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.563191 (+857345us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.577736 (+ 14545us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.577881 (+ 145us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:23.901660 739 rpcz_store.cc:276] 0905 19:52:22.589011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.613403 (+ 24392us) service_pool.cc:225] Handling call
0905 19:52:23.279332 (+665929us) ts_tablet_manager.cc:663] Acquired tablet manager lock
0905 19:52:23.358826 (+ 79494us) ts_tablet_manager.cc:676] Creating new metadata...
0905 19:52:23.592500 (+233674us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.615057 (+ 22557us) tablet_replica.cc:206] Creating consensus instance
0905 19:52:23.615152 (+ 95us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":634}
I20250905 19:52:23.752089 1803 raft_consensus.cc:491] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.904129 1803 raft_consensus.cc:513] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.905033 1803 leader_election.cc:290] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:23.911729 848 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.912314 848 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.912467 849 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:23.912593 849 leader_election.cc:304] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.913004 1658 raft_consensus.cc:2747] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:23.916216 849 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
I20250905 19:52:23.916347 849 leader_election.cc:304] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:23.917961 1798 raft_consensus.cc:491] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.918045 1809 raft_consensus.cc:491] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:23.918157 1798 raft_consensus.cc:513] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.918905 1798 leader_election.cc:290] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:23.919430 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:23.920049 1799 leader_election.cc:290] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:23.918203 1809 raft_consensus.cc:513] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.920629 1809 leader_election.cc:290] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:23.921483 1169 tablet_bootstrap.cc:492] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:23.922765 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e10b4e82c4bc429b9a42db39c03a5267 consensus state has no leader
I20250905 19:52:23.922946 452 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52586 (request call id 515) took 1095 ms. Trace:
I20250905 19:52:23.923210 1169 tablet_bootstrap.cc:654] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:23.923044 452 rpcz_store.cc:276] 0905 19:52:22.827568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.419536 (+591968us) service_pool.cc:225] Handling call
0905 19:52:23.423164 (+ 3628us) spinlock_profiling.cc:243] Waited 3.42 ms on lock 0x616000041254. stack: 00007f8bd3ea2dc6 00007f8bd3ea2b80 00007f8bd2e8a3ad 0000000000972d94 0000000000972d5d 00007f8be2ca914d 00007f8be2ca9243 00007f8bdf968941 00007f8be74f3a7d 00007f8be767f146 00007f8bde4b8126 00007f8bde4b7e55 00007f8bdac8c0be 00007f8bdac8af9e 00007f8bdac911d5 00007f8bdac91aff
0905 19:52:23.423366 (+ 202us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:23.423547 (+ 181us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:23.423691 (+ 144us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:23.423751 (+ 60us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:23.922549 (+498798us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:23.922888 (+ 339us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:23.533286 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:23.533415 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:23.533420 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:23.533422 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:23.533528 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:23.533543 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:23.533548 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:23.533608 (+ 60us) tablet.cc:699] Row locks acquired
0905 19:52:23.533610 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:23.533643 (+ 33us) write_op.cc:270] Start()
0905 19:52:23.533672 (+ 29us) write_op.cc:276] Timestamp: P: 1757101943533636 usec, L: 0
0905 19:52:23.533675 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:23.533832 (+ 157us) log.cc:844] Serialized 405 byte log entry
0905 19:52:23.920814 (+386982us) op_driver.cc:464] REPLICATION: finished
0905 19:52:23.921142 (+ 328us) write_op.cc:301] APPLY: starting
0905 19:52:23.921162 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:23.921206 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:23.921239 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:23.921291 (+ 52us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:23.921324 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:23.921330 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:23.921421 (+ 91us) log.cc:844] Serialized 23 byte log entry
0905 19:52:23.921493 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:23.921514 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:23.921522 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:23.921596 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:23.921599 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":7864192,"child_traces":[["op",{"apply.queue_time_us":268,"apply.run_cpu_time_us":449,"apply.run_wall_time_us":450,"num_ops":1,"prepare.queue_time_us":109167,"prepare.run_cpu_time_us":651,"prepare.run_wall_time_us":8853,"replication_time_us":387106}]]}
I20250905 19:52:23.925388 454 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52586 (request call id 516) took 1097 ms. Trace:
I20250905 19:52:23.925551 454 rpcz_store.cc:276] 0905 19:52:22.827639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.454899 (+627260us) service_pool.cc:225] Handling call
0905 19:52:23.458373 (+ 3474us) spinlock_profiling.cc:243] Waited 3.21 ms on lock 0x616000041254. stack: 00007f8bd3ea2dc6 00007f8bd3ea2b80 00007f8bd2e8a3ad 0000000000972d94 0000000000972d5d 00007f8be2ca914d 00007f8be2ca9243 00007f8bdf968941 00007f8be74f3a7d 00007f8be767ceb3 00007f8bde4b47a6 00007f8bde4b44d5 00007f8bdac8c0be 00007f8bdac8af9e 00007f8bdac911d5 00007f8bdac91aff
0905 19:52:23.459110 (+ 737us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:23.459645 (+ 535us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:23.925066 (+465421us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:23.925192 (+ 126us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:23.925311 (+ 119us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:23.542213 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:23.552148 (+ 9935us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:23.552155 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:23.552157 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:23.552241 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:23.552251 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:23.552256 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:23.552531 (+ 275us) tablet.cc:699] Row locks acquired
0905 19:52:23.552534 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:23.552574 (+ 40us) write_op.cc:270] Start()
0905 19:52:23.552606 (+ 32us) write_op.cc:276] Timestamp: P: 1757101943552564 usec, L: 0
0905 19:52:23.552609 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:23.552788 (+ 179us) log.cc:844] Serialized 433 byte log entry
0905 19:52:23.920889 (+368101us) op_driver.cc:464] REPLICATION: finished
0905 19:52:23.921669 (+ 780us) write_op.cc:301] APPLY: starting
0905 19:52:23.921680 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:23.921712 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:23.921734 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:23.921853 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:23.921884 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:23.921890 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:23.921951 (+ 61us) log.cc:844] Serialized 29 byte log entry
0905 19:52:23.921993 (+ 42us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:23.922007 (+ 14us) write_op.cc:454] Released schema lock
0905 19:52:23.922013 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:23.922079 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:23.922083 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":7390464,"child_traces":[["op",{"apply.queue_time_us":748,"apply.run_cpu_time_us":402,"apply.run_wall_time_us":402,"num_ops":2,"prepare.queue_time_us":82040,"prepare.run_cpu_time_us":751,"prepare.run_wall_time_us":10660,"replication_time_us":368263}]]}
W20250905 19:52:23.927037 446 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 239539ac19124de192cadae494720a2b consensus state has no leader
I20250905 19:52:23.927201 446 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:52586 (request call id 517) took 1032 ms. Trace:
I20250905 19:52:23.927301 446 rpcz_store.cc:276] 0905 19:52:22.895156 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.470450 (+575294us) service_pool.cc:225] Handling call
0905 19:52:23.476061 (+ 5611us) spinlock_profiling.cc:243] Waited 5.35 ms on lock 0x616000041254. stack: 00007f8bd3ea2dc6 00007f8bd3ea2b80 00007f8bd2e8a3ad 0000000000972d94 0000000000972d5d 00007f8be2ca914d 00007f8be2ca9243 00007f8bdf968941 00007f8be74f3a7d 00007f8be767f146 00007f8bde4b8126 00007f8bde4b7e55 00007f8bdac8c0be 00007f8bdac8af9e 00007f8bdac911d5 00007f8bdac91aff
0905 19:52:23.476324 (+ 263us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:23.476529 (+ 205us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:23.476694 (+ 165us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:23.476779 (+ 85us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:23.926818 (+450039us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:23.927153 (+ 335us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:23.552896 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:23.552984 (+ 88us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:23.552989 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:23.552993 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:23.553090 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:23.553101 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:23.553106 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:23.553146 (+ 40us) tablet.cc:699] Row locks acquired
0905 19:52:23.553148 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:23.553175 (+ 27us) write_op.cc:270] Start()
0905 19:52:23.553199 (+ 24us) write_op.cc:276] Timestamp: P: 1757101943553168 usec, L: 0
0905 19:52:23.553202 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:23.553331 (+ 129us) log.cc:844] Serialized 405 byte log entry
0905 19:52:23.920941 (+367610us) op_driver.cc:464] REPLICATION: finished
0905 19:52:23.922149 (+ 1208us) write_op.cc:301] APPLY: starting
0905 19:52:23.922159 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:23.922189 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:23.922218 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:23.922256 (+ 38us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:23.922285 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:23.922295 (+ 10us) write_op.cc:312] APPLY: finished
0905 19:52:23.922349 (+ 54us) log.cc:844] Serialized 23 byte log entry
0905 19:52:23.922398 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:23.922413 (+ 15us) write_op.cc:454] Released schema lock
0905 19:52:23.922420 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":12315392,"child_traces":[["op",{"apply.queue_time_us":1168,"apply.run_cpu_time_us":325,"apply.run_wall_time_us":325,"num_ops":1,"prepare.queue_time_us":75695,"prepare.run_cpu_time_us":497,"prepare.run_wall_time_us":496,"replication_time_us":367724,"threads_started":1}]]}
I20250905 19:52:23.930732 1673 raft_consensus.cc:2747] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:23.931206 1769 tablet.cc:1722] T 9f732ae0627c4da5b1691cde9b4d5562 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:23.934741 1728 tablet.cc:1722] T 9f732ae0627c4da5b1691cde9b4d5562 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:23.946308 1185 ts_tablet_manager.cc:1905] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.933599 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:23.947400 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_2c0cad91758c4851893e8a5919f5d44c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.947619 986 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.948064 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_2c0cad91758c4851893e8a5919f5d44c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.948680 1184 ts_tablet_manager.cc:1918] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:23.948877 1184 log.cc:1199] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:23.949255 1184 ts_tablet_manager.cc:1939] T 8ed6109bdf8f4caf8c223a42401b94ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:23.949203 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13d282e1dc54d1faa2882c9b39d17cb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:23.950008 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 318) took 2987 ms. Trace:
I20250905 19:52:23.950138 1184 rpcz_store.cc:276] 0905 19:52:20.962833 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.962877 (+ 44us) service_pool.cc:225] Handling call
0905 19:52:23.524760 (+2561883us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6109bdf8f4caf8c223a42401b94ab
0905 19:52:23.524769 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.933039 (+408270us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.948539 (+ 15500us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.949949 (+ 1410us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2561694}
W20250905 19:52:23.951057 986 leader_election.cc:343] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:23.951442 451 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ea6df8634a314ceabcf3fd004bf7e241 consensus state has no leader
I20250905 19:52:23.951905 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_01c7456a16814955b2c2f3890ecb0d01"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.952476 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_01c7456a16814955b2c2f3890ecb0d01 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.954422 446 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_f1ff5bb8b0734e03bca82540e14ee739"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.954905 446 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1ff5bb8b0734e03bca82540e14ee739 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.958287 454 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4fe8c2a1caf340069ff229767f87e772"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:23.958770 454 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4fe8c2a1caf340069ff229767f87e772 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:23.965749 452 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 1 got 0
I20250905 19:52:23.970500 1184 tablet_replica.cc:331] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:23.970738 1184 raft_consensus.cc:2241] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:23.970947 1184 raft_consensus.cc:2270] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.971560 1184 ts_tablet_manager.cc:1905] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:23.971674 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8ed6109bdf8f4caf8c223a42401b94ab (table default.Table_68f98d1c0de04ff9a6ad5188ab275ec9 [id=711f36b2be7c4a368fcb3106e691b133]) successfully deleted
I20250905 19:52:23.982635 1171 tablet_bootstrap.cc:492] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:23.982805 1171 ts_tablet_manager.cc:1397] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.029s
I20250905 19:52:23.983736 1171 raft_consensus.cc:357] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.983819 888 tablet_service.cc:1468] Processing CreateTablet for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (DEFAULT_TABLE table=default.Table_ac8e97a442114459a1ff85613d8651f9 [id=10e097461e0f4f3bbf8a59753f69e913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.983937 1171 raft_consensus.cc:383] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:23.984053 1171 raft_consensus.cc:738] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.984297 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a860ab3f5ce54bd2b75bb71730a3c6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.984398 1171 consensus_queue.cc:260] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.984576 889 tablet_service.cc:1468] Processing CreateTablet for tablet e63b27ff4e434d76a8280f90c22f4082 (DEFAULT_TABLE table=default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a [id=1598bab0cf8f41e0a4f7ee643cfeeaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.984980 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63b27ff4e434d76a8280f90c22f4082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.984957 1171 ts_tablet_manager.cc:1428] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:23.985391 1171 tablet_bootstrap.cc:492] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:23.987285 1171 tablet_bootstrap.cc:654] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:23.988322 885 tablet_service.cc:1468] Processing CreateTablet for tablet c77dd068ea5c495ca8aa0e7a6823d792 (DEFAULT_TABLE table=default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.988826 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c77dd068ea5c495ca8aa0e7a6823d792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.992324 890 tablet_service.cc:1468] Processing CreateTablet for tablet 180437d1534946619384307ddcb7c937 (DEFAULT_TABLE table=default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.993440 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180437d1534946619384307ddcb7c937. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.996343 880 tablet_service.cc:1468] Processing CreateTablet for tablet 5b4f9e12781a46f19b563746a70b59e4 (DEFAULT_TABLE table=default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.996829 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4f9e12781a46f19b563746a70b59e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.998054 1171 tablet_bootstrap.cc:492] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:23.998221 1171 ts_tablet_manager.cc:1397] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.004s
I20250905 19:52:23.999200 1171 raft_consensus.cc:357] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:23.999392 1171 raft_consensus.cc:383] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:23.999456 1171 raft_consensus.cc:738] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:23.999723 1171 consensus_queue.cc:260] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.000262 1171 ts_tablet_manager.cc:1428] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:24.009084 1169 tablet_bootstrap.cc:492] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:23.928107 1708 tablet.cc:1722] T 59c13a5b598a4597931ddf5d476aec85 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:24.038862 452 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]). Expected version 1 got 0
I20250905 19:52:24.009650 1022 tablet_service.cc:1468] Processing CreateTablet for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (DEFAULT_TABLE table=default.Table_ac8e97a442114459a1ff85613d8651f9 [id=10e097461e0f4f3bbf8a59753f69e913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.000288 892 tablet_service.cc:1468] Processing CreateTablet for tablet a551e641a19f4d3f924fb176c4a25ca7 (DEFAULT_TABLE table=default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.000602 1171 tablet_bootstrap.cc:492] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:24.229192 452 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:24.229326 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a860ab3f5ce54bd2b75bb71730a3c6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.229362 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a551e641a19f4d3f924fb176c4a25ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.229476 452 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:24.229702 452 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac]). Expected version 2 got 0
I20250905 19:52:24.231156 1171 tablet_bootstrap.cc:654] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.000680 894 tablet_service.cc:1468] Processing CreateTablet for tablet 65368306f86845008d4d486023e1fc29 (DEFAULT_TABLE table=default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.232462 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65368306f86845008d4d486023e1fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.232513 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_ac8e97a442114459a1ff85613d8651f9" } new_table_name: "default.Table_9bda3d1d061946baa90945381c53758b" modify_external_catalogs: true
I20250905 19:52:23.930928 1170 tablet_bootstrap.cc:654] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:23.942894 1186 ts_tablet_manager.cc:1918] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:23.975440 739 tablet_service.cc:1468] Processing CreateTablet for tablet d59da39e7bcf497a93fb9174328c1814 (DEFAULT_TABLE table=default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.233949 1186 log.cc:1199] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/66df0522aa2842f68aa824571fa61b05
I20250905 19:52:24.234210 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d59da39e7bcf497a93fb9174328c1814. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.013461 1032 tablet_service.cc:1468] Processing CreateTablet for tablet c77dd068ea5c495ca8aa0e7a6823d792 (DEFAULT_TABLE table=default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.234448 1186 ts_tablet_manager.cc:1939] T 66df0522aa2842f68aa824571fa61b05 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:24.234706 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c77dd068ea5c495ca8aa0e7a6823d792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.235289 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48712 (request call id 388) took 1602 ms. Trace:
I20250905 19:52:24.235416 1186 rpcz_store.cc:276] 0905 19:52:22.632515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.642057 (+ 9542us) service_pool.cc:225] Handling call
0905 19:52:23.893814 (+1251757us) ts_tablet_manager.cc:1145] Deleting tablet 66df0522aa2842f68aa824571fa61b05
0905 19:52:23.893822 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.930169 (+ 36347us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:23.942751 (+ 12582us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.235245 (+292494us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":668365}
I20250905 19:52:24.019786 1185 ts_tablet_manager.cc:1918] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:24.236089 1185 log.cc:1199] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:24.236531 1185 ts_tablet_manager.cc:1939] T 19e4edddafcd4d22a0e0ae66b0b953e0 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.098135 889 tablet_service.cc:1468] Processing CreateTablet for tablet 923ad8e40a07409e8f6a0152a0b2ef75 (DEFAULT_TABLE table=default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.237169 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 923ad8e40a07409e8f6a0152a0b2ef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.237370 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 309) took 3478 ms. Trace:
I20250905 19:52:24.237474 1185 rpcz_store.cc:276] 0905 19:52:20.758831 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.760763 (+ 1932us) service_pool.cc:225] Handling call
0905 19:52:23.887287 (+3126524us) ts_tablet_manager.cc:1145] Deleting tablet 19e4edddafcd4d22a0e0ae66b0b953e0
0905 19:52:23.887297 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:23.959684 (+ 72387us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.019681 (+ 59997us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.237312 (+217631us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3126176}
I20250905 19:52:24.238008 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 311) took 3445 ms. Trace:
I20250905 19:52:24.238103 1185 rpcz_store.cc:276] 0905 19:52:20.792563 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.807016 (+ 14453us) service_pool.cc:225] Handling call
0905 19:52:24.237835 (+3430819us) ts_tablet_manager.cc:1145] Deleting tablet fd793bce71e344f18bb177a53ab1bfc1
0905 19:52:24.237844 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.237948 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3427862}
I20250905 19:52:24.100849 880 tablet_service.cc:1468] Processing CreateTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 (DEFAULT_TABLE table=default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.106510 888 tablet_service.cc:1468] Processing CreateTablet for tablet 74eb2632a6414e52b9ad67f53c182679 (DEFAULT_TABLE table=default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.239174 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 19e4edddafcd4d22a0e0ae66b0b953e0 (table default.Table_ff5415e10957422498084d7730365989 [id=bd0b3b55bf3648b7a45b7aef904be570]) successfully deleted
I20250905 19:52:24.155268 885 tablet_service.cc:1468] Processing CreateTablet for tablet f6f29ac5e68d4d8da8f3496fefe695ec (DEFAULT_TABLE table=default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.156450 890 tablet_service.cc:1468] Processing CreateTablet for tablet 5aff2acc3f8245e1bca41f971f8e74ce (DEFAULT_TABLE table=default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:24.239503 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet fd793bce71e344f18bb177a53ab1bfc1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd793bce71e344f18bb177a53ab1bfc1 already in progress: creating tablet
I20250905 19:52:24.239689 1185 tablet_replica.cc:331] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.076304 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_33f56154c6394adf9d27effde71d7363"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:24.240013 880 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69a41cee5ff4f8f9bccaa513c096118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.240358 885 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f29ac5e68d4d8da8f3496fefe695ec. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:24.240540 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33f56154c6394adf9d27effde71d7363 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.240998 888 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eb2632a6414e52b9ad67f53c182679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.240975 1185 raft_consensus.cc:2241] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:24.241344 1185 pending_rounds.cc:70] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:24.241418 1185 pending_rounds.cc:77] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089556690067456 op_type: NO_OP noop_request { }
I20250905 19:52:24.241593 1185 raft_consensus.cc:2887] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:24.157763 875 tablet_service.cc:1468] Processing CreateTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (DEFAULT_TABLE table=default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.241700 1185 raft_consensus.cc:2270] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:23.976205 745 tablet_service.cc:1468] Processing CreateTablet for tablet 862214a31059407baefef8a390aaea69 (DEFAULT_TABLE table=default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.242138 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 862214a31059407baefef8a390aaea69. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.159785 881 tablet_service.cc:1468] Processing CreateTablet for tablet f34f0b29d6104cac8ab35309fd1edae8 (DEFAULT_TABLE table=default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.160161 877 tablet_service.cc:1468] Processing CreateTablet for tablet c870e689af254fb0acb15330577e5c42 (DEFAULT_TABLE table=default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.160974 891 tablet_service.cc:1468] Processing CreateTablet for tablet b1f46261822442e0a961d2d63bdb35c8 (DEFAULT_TABLE table=default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.242699 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5428f20ca14fdf88a3a52f7b4d1e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.242915 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f46261822442e0a961d2d63bdb35c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.243119 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34f0b29d6104cac8ab35309fd1edae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.245016 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c870e689af254fb0acb15330577e5c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.245950 890 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aff2acc3f8245e1bca41f971f8e74ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.038618 1169 ts_tablet_manager.cc:1397] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.155s user 0.003s sys 0.003s
I20250905 19:52:24.247021 1169 raft_consensus.cc:357] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.247221 1169 raft_consensus.cc:383] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.247318 1169 raft_consensus.cc:738] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.267500 1704 raft_consensus.cc:491] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.247519 1169 consensus_queue.cc:260] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.360110 1704 raft_consensus.cc:513] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.360415 1169 ts_tablet_manager.cc:1428] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.114s user 0.002s sys 0.000s
I20250905 19:52:24.360754 1169 tablet_bootstrap.cc:492] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.045550 1030 tablet_service.cc:1468] Processing CreateTablet for tablet e63b27ff4e434d76a8280f90c22f4082 (DEFAULT_TABLE table=default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a [id=1598bab0cf8f41e0a4f7ee643cfeeaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.360821 1704 leader_election.cc:290] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.269903 1185 ts_tablet_manager.cc:1905] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.361203 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63b27ff4e434d76a8280f90c22f4082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.362629 1169 tablet_bootstrap.cc:654] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.046015 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 180437d1534946619384307ddcb7c937 (DEFAULT_TABLE table=default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.364315 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180437d1534946619384307ddcb7c937. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.276037 1171 tablet_bootstrap.cc:492] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:24.046610 1025 tablet_service.cc:1468] Processing CreateTablet for tablet a551e641a19f4d3f924fb176c4a25ca7 (DEFAULT_TABLE table=default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.367324 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a551e641a19f4d3f924fb176c4a25ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:23.976686 751 tablet_service.cc:1468] Processing CreateTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 (DEFAULT_TABLE table=default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.977171 754 tablet_service.cc:1468] Processing CreateTablet for tablet af10abf0a5b34b3bbfea20e5c2005006 (DEFAULT_TABLE table=default.Table_7b30c5a5c660490191d00c27224d516f [id=8213b623d8414c62874496694b1515fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.977548 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:23.977478 750 tablet_service.cc:1468] Processing CreateTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 (DEFAULT_TABLE table=default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.977854 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.052737 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 5b4f9e12781a46f19b563746a70b59e4 (DEFAULT_TABLE table=default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:23.978158 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4204f61929498188c18529148a8d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:23.978461 743 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:23.978740 756 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:23.979032 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 77cbdda0e2d641bd8092f00925bb9a9c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:23.979367 753 tablet_service.cc:1515] Processing DeleteTablet for tablet e6f837fb7293474b8ad357f52e42e42e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.376552 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4f9e12781a46f19b563746a70b59e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.052949 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 65368306f86845008d4d486023e1fc29 (DEFAULT_TABLE table=default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.059921 1184 ts_tablet_manager.cc:1918] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:23.980176 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 76a5c85dbda140d882a5a63a13aaabd7 with delete_type TABLET_DATA_DELETED (replaced by tablet f28e403dc2f6471ebea1805fd843cfb1) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:23.980402 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 57f851a6eecc4a41806bdaa19cc5afab with delete_type TABLET_DATA_DELETED (replaced by tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.380479 1184 log.cc:1199] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:23.980566 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 7edcf57f7753449d8a5fcdb6f4f1dc21 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.026898 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13d282e1dc54d1faa2882c9b39d17cb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.076738 451 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_19682dbce0bc4480b82d39235ba7f55b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:24.380896 1184 ts_tablet_manager.cc:1939] T 302053e1611b4e22a9a2e7b60ae68f27 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:24.381242 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 319) took 3418 ms. Trace:
W20250905 19:52:24.381383 451 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_19682dbce0bc4480b82d39235ba7f55b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.381374 1184 rpcz_store.cc:276] 0905 19:52:20.963198 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.963240 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:23.970396 (+3007156us) ts_tablet_manager.cc:1145] Deleting tablet 302053e1611b4e22a9a2e7b60ae68f27
0905 19:52:23.970409 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.050377 (+ 79968us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.059751 (+ 9374us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.381206 (+321455us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3006949}
I20250905 19:52:24.381821 1184 tablet_replica.cc:331] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:24.382066 1184 raft_consensus.cc:2241] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.382256 1184 raft_consensus.cc:2270] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.382859 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65368306f86845008d4d486023e1fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.382891 1184 ts_tablet_manager.cc:1905] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.099247 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 923ad8e40a07409e8f6a0152a0b2ef75 (DEFAULT_TABLE table=default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.384310 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 923ad8e40a07409e8f6a0152a0b2ef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.026985 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:23.979862 712 leader_election.cc:343] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 77cbdda0e2d641bd8092f00925bb9a9c
W20250905 19:52:24.192489 698 kernel_stack_watchdog.cc:198] Thread 712 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 147ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7fb0da75e980 (unknown) at ??:0
@ 0x7fb0da7589c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7fb0d4593904 google::LogMessage::Flush() at ??:0
@ 0x7fb0d4593a11 google::LogMessage::~LogMessage() at ??:0
@ 0x7fb0e2d04973 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7fb0e2d074ea kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7fb0e2d072ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fb0de02ec4f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7fb0de02e82e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7fb0de01a860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7fb0de01adac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7fb0ddfb62aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7fb0ddfb4c66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7fb0ddfc7135 ev::base<>::method_thunk<>() at ??:0
I20250905 19:52:24.388135 712 leader_election.cc:304] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:24.374366 1171 ts_tablet_manager.cc:1397] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.374s user 0.004s sys 0.000s
W20250905 19:52:24.326234 848 consensus_peers.cc:489] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 66df0522aa2842f68aa824571fa61b05. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:24.388373 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 62227.6us
I20250905 19:52:24.077056 446 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1c6571236cd7434d98aec149d06dc6b8" } modify_external_catalogs: true
I20250905 19:52:24.388623 446 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_1c6571236cd7434d98aec149d06dc6b8" } modify_external_catalogs: true
W20250905 19:52:24.371327 849 leader_election.cc:343] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 74a7e7d9874e4d26bda3ab24a6e24adc
W20250905 19:52:24.388770 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 408921us
I20250905 19:52:24.389267 1704 raft_consensus.cc:491] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.389406 1704 raft_consensus.cc:513] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.290926 878 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbd471695e748c9831f9b0f6795aea7 (DEFAULT_TABLE table=default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.390017 1704 leader_election.cc:290] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.390409 747 tablet_service.cc:1468] Processing CreateTablet for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (DEFAULT_TABLE table=default.Table_ac8e97a442114459a1ff85613d8651f9 [id=10e097461e0f4f3bbf8a59753f69e913]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.390856 740 tablet_service.cc:1468] Processing CreateTablet for tablet c77dd068ea5c495ca8aa0e7a6823d792 (DEFAULT_TABLE table=default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.391397 1849 raft_consensus.cc:2747] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.391476 741 tablet_service.cc:1468] Processing CreateTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (DEFAULT_TABLE table=default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.102659 1031 tablet_service.cc:1468] Processing CreateTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 (DEFAULT_TABLE table=default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.391759 441 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 302053e1611b4e22a9a2e7b60ae68f27 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:24.391981 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69a41cee5ff4f8f9bccaa513c096118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.391927 752 tablet_service.cc:1468] Processing CreateTablet for tablet f34f0b29d6104cac8ab35309fd1edae8 (DEFAULT_TABLE table=default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.392256 753 tablet_service.cc:1468] Processing CreateTablet for tablet 180437d1534946619384307ddcb7c937 (DEFAULT_TABLE table=default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.392380 748 tablet_service.cc:1468] Processing CreateTablet for tablet c870e689af254fb0acb15330577e5c42 (DEFAULT_TABLE table=default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.392675 746 tablet_service.cc:1468] Processing CreateTablet for tablet 5b4f9e12781a46f19b563746a70b59e4 (DEFAULT_TABLE table=default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.392793 749 tablet_service.cc:1468] Processing CreateTablet for tablet 74eb2632a6414e52b9ad67f53c182679 (DEFAULT_TABLE table=default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.393038 756 tablet_service.cc:1468] Processing CreateTablet for tablet a551e641a19f4d3f924fb176c4a25ca7 (DEFAULT_TABLE table=default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.393239 737 tablet_service.cc:1468] Processing CreateTablet for tablet e63b27ff4e434d76a8280f90c22f4082 (DEFAULT_TABLE table=default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a [id=1598bab0cf8f41e0a4f7ee643cfeeaab]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.393426 743 tablet_service.cc:1468] Processing CreateTablet for tablet 65368306f86845008d4d486023e1fc29 (DEFAULT_TABLE table=default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.393702 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d1a11777cf2d4ea9a5cdf792f73aff76. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.393821 755 tablet_service.cc:1468] Processing CreateTablet for tablet 923ad8e40a07409e8f6a0152a0b2ef75 (DEFAULT_TABLE table=default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.394263 744 tablet_service.cc:1468] Processing CreateTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 (DEFAULT_TABLE table=default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.394681 739 tablet_service.cc:1468] Processing CreateTablet for tablet f6f29ac5e68d4d8da8f3496fefe695ec (DEFAULT_TABLE table=default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.395100 745 tablet_service.cc:1468] Processing CreateTablet for tablet 5aff2acc3f8245e1bca41f971f8e74ce (DEFAULT_TABLE table=default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.270490 1170 tablet_bootstrap.cc:492] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:24.390038 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbd471695e748c9831f9b0f6795aea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.396157 1170 ts_tablet_manager.cc:1397] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.525s user 0.005s sys 0.006s
I20250905 19:52:24.396632 738 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbd471695e748c9831f9b0f6795aea7 (DEFAULT_TABLE table=default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.397015 1170 raft_consensus.cc:357] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.397256 1170 raft_consensus.cc:383] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.397321 1170 raft_consensus.cc:738] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.397693 1170 consensus_queue.cc:260] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.077481 454 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_aa894f644dbe444bbe04d396b9ceb056" } modify_external_catalogs: true
I20250905 19:52:24.398258 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 65368306f86845008d4d486023e1fc29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.398466 454 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_aa894f644dbe444bbe04d396b9ceb056" } modify_external_catalogs: true
I20250905 19:52:24.106063 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 74eb2632a6414e52b9ad67f53c182679 (DEFAULT_TABLE table=default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.400128 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eb2632a6414e52b9ad67f53c182679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.078706 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 00ac56c449ba4b7b8dbd5e87120799d9
I20250905 19:52:24.404246 755 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 923ad8e40a07409e8f6a0152a0b2ef75. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.404739 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 8d38452d3a0641ffa4af5d935f8ad9f0 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:24.405443 453 master_service.cc:948] ReplaceTablet: received request to replace tablet b0702063ec1c4f6fba58e6d4f3912212 from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:24.405529 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:24.405822 453 master_service.cc:948] ReplaceTablet: received request to replace tablet 5b0d4be788f841c2860f5dd65275b623 from {username='slave'} at 127.0.0.1:52586
W20250905 19:52:24.406116 849 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:24.155807 1033 tablet_service.cc:1468] Processing CreateTablet for tablet f6f29ac5e68d4d8da8f3496fefe695ec (DEFAULT_TABLE table=default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.407215 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f29ac5e68d4d8da8f3496fefe695ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.408460 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c69a41cee5ff4f8f9bccaa513c096118. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.157155 1021 tablet_service.cc:1468] Processing CreateTablet for tablet 5aff2acc3f8245e1bca41f971f8e74ce (DEFAULT_TABLE table=default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.158993 1024 tablet_service.cc:1468] Processing CreateTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (DEFAULT_TABLE table=default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.159373 1026 tablet_service.cc:1468] Processing CreateTablet for tablet f34f0b29d6104cac8ab35309fd1edae8 (DEFAULT_TABLE table=default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.160526 1028 tablet_service.cc:1468] Processing CreateTablet for tablet c870e689af254fb0acb15330577e5c42 (DEFAULT_TABLE table=default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.162168 1016 tablet_service.cc:1468] Processing CreateTablet for tablet b1f46261822442e0a961d2d63bdb35c8 (DEFAULT_TABLE table=default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.038560 1853 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.409803 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.409945 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aff2acc3f8245e1bca41f971f8e74ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.409982 1853 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.410742 1853 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.290493 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 5cbd471695e748c9831f9b0f6795aea7 (DEFAULT_TABLE table=default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:24.048923 984 leader_election.cc:343] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.410993 984 leader_election.cc:304] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:24.411330 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 385248us
I20250905 19:52:24.079241 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 14d5eb8bec074d2c8766fe7b159031e5
I20250905 19:52:24.079464 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a1b4989f17454aa58c05727b797c2009
W20250905 19:52:24.412424 984 leader_election.cc:343] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.412559 984 leader_election.cc:304] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.079732 447 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_834cb637abd74d7694ba9e3f8e4ed84a" } new_table_name: "default.Table_9da2eedb50e64356ba6a49d93bafbe28" modify_external_catalogs: true
I20250905 19:52:24.412616 455 master_service.cc:948] ReplaceTablet: received request to replace tablet 8503f6918ffb44efa36214633c93e31d from {username='slave'} at 127.0.0.1:52586
W20250905 19:52:24.412901 984 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.413043 984 leader_election.cc:304] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:24.413218 984 raft_consensus.cc:2684] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250905 19:52:24.359294 970 kernel_stack_watchdog.cc:198] Thread 984 stuck at /home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/rpc/outbound_call.cc:307 for 276ms:
Kernel stack:
(could not read kernel stack)
User stack:
@ 0x7f73f5b49980 (unknown) at ??:0
@ 0x7f73f5b439c6 __GI___pthread_rwlock_wrlock at ??:0
@ 0x7f73ef97e904 google::LogMessage::Flush() at ??:0
@ 0x7f73ef97ea11 google::LogMessage::~LogMessage() at ??:0
@ 0x7f73fe0ef973 kudu::consensus::LeaderElection::VoteResponseRpcCallback() at ??:0
@ 0x7f73fe0f24ea kudu::consensus::LeaderElection::Run()::$_0::operator()() at ??:0
@ 0x7f73fe0f22ae std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f73f9419c4f kudu::rpc::Proxy::AsyncRequest()::$_1::operator()() at ??:0
@ 0x7f73f941982e std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa774dd std::function<>::operator()() at ??:?
@ 0x7f73f9405860 kudu::rpc::OutboundCall::CallCallback() at ??:0
@ 0x7f73f9405dac kudu::rpc::OutboundCall::SetResponse() at ??:0
@ 0x7f73f93a12aa kudu::rpc::Connection::HandleCallResponse() at ??:0
@ 0x7f73f939fc66 kudu::rpc::Connection::ReadHandler() at ??:0
@ 0x7f73f93b2135 ev::base<>::method_thunk<>() at ??:0
I20250905 19:52:24.248332 452 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75] alter complete (version 2)
I20250905 19:52:24.413893 1799 raft_consensus.cc:2747] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.414068 1799 raft_consensus.cc:491] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.414186 1799 raft_consensus.cc:513] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.396178 742 tablet_service.cc:1468] Processing CreateTablet for tablet b1f46261822442e0a961d2d63bdb35c8 (DEFAULT_TABLE table=default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:24.414731 1799 leader_election.cc:290] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.415041 1799 raft_consensus.cc:2747] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.415500 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34f0b29d6104cac8ab35309fd1edae8. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:24.248680 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet a860ab3f5ce54bd2b75bb71730a3c6e6 consensus state has no leader
I20250905 19:52:24.265813 450 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_cb111dc4aa3946e8a89529d1cd623c08 [id=5ac15645cb4d4730a4b4464a85299bed] alter complete (version 5)
W20250905 19:52:24.309003 440 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet a860ab3f5ce54bd2b75bb71730a3c6e6 consensus state has no leader
I20250905 19:52:24.416728 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.416966 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.417629 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet cf2881d228964cd3bef040a06045d8d2 (table default.Table_175c78c56c0149e8919f8220e87e3d4b [id=1656bb34c67340f4b3793d8ffdf3a16c]) successfully deleted
I20250905 19:52:24.417754 1171 raft_consensus.cc:357] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.417929 1171 raft_consensus.cc:383] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.418004 1171 raft_consensus.cc:738] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.418028 440 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 66df0522aa2842f68aa824571fa61b05 (table default.Table_f834b7f48d2d450ea637f8cfe2f9c3c3 [id=53917d92e1214845a54503f40686b0e7]) successfully deleted
I20250905 19:52:24.418221 1171 consensus_queue.cc:260] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.418725 1171 ts_tablet_manager.cc:1428] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.030s user 0.002s sys 0.000s
I20250905 19:52:24.418943 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c870e689af254fb0acb15330577e5c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.418941 449 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_7b30c5a5c660490191d00c27224d516f" } new_table_name: "default.Table_b9646298e8214aa184d57fe2b66b6b0e" modify_external_catalogs: true
I20250905 19:52:24.419064 1171 tablet_bootstrap.cc:492] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:24.420387 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_5bf03647f71d46a8b3d3f0b5bc385154"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:24.420938 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5bf03647f71d46a8b3d3f0b5bc385154 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.412616 448 master_service.cc:948] ReplaceTablet: received request to replace tablet ba161282480e4736889559a2e37b195f from {username='slave'} at 127.0.0.1:52586
I20250905 19:52:24.422104 452 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_51169c758cd646cd91f3bda396598e10" } new_table_name: "default.Table_d6d822b5cec5446aa8b1db2a98074ef6" modify_external_catalogs: true
I20250905 19:52:24.423206 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.423542 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f46261822442e0a961d2d63bdb35c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.398296 1170 ts_tablet_manager.cc:1428] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:24.428295 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbd471695e748c9831f9b0f6795aea7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:24.430399 848 consensus_peers.cc:489] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:24.432197 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5428f20ca14fdf88a3a52f7b4d1e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.432299 1584 consensus_queue.cc:1035] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:24.434566 446 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:24.437659 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:24.438055 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:24.438727 889 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.439553 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 1 got 0
I20250905 19:52:24.439807 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:24.441059 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_72aac5d5ddba4d23a99e5940bad14bb3 [id=3118b283e4194c5fa5c5a7e270d14e49]). Expected version 1 got 0
I20250905 19:52:24.441267 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:24.441447 446 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:24.379887 1185 ts_tablet_manager.cc:1918] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:24.443727 1185 log.cc:1199] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:24.444373 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:24.444412 986 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:24.444677 986 leader_election.cc:343] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.444852 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:24.446650 454 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet a8b6823580cb4040b49ed0a49b748965
I20250905 19:52:24.446837 1171 tablet_bootstrap.cc:654] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.447467 454 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_72aac5d5ddba4d23a99e5940bad14bb3" } new_table_name: "default.Table_c30c09bc294d465da486648091481ed1" modify_external_catalogs: true
W20250905 19:52:24.448228 447 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 415508e72ee34bc391c4369b5f8a2f48 consensus state has no leader
I20250905 19:52:24.448858 447 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_af420aac7cc249339982b49a7f3942a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:24.449517 447 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af420aac7cc249339982b49a7f3942a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.450493 1185 ts_tablet_manager.cc:1939] T 4c76077e4adf45a7adc736b0e69032eb P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:24.450913 449 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet af10abf0a5b34b3bbfea20e5c2005006 consensus state has no leader
I20250905 19:52:24.451244 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6ec4f72266c7447da59b8544cfa98cd6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:24.451710 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 312) took 3539 ms. Trace:
W20250905 19:52:24.451762 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ec4f72266c7447da59b8544cfa98cd6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.451802 1185 rpcz_store.cc:276] 0905 19:52:20.912636 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.912702 (+ 66us) service_pool.cc:225] Handling call
0905 19:52:24.238316 (+3325614us) ts_tablet_manager.cc:1145] Deleting tablet 4c76077e4adf45a7adc736b0e69032eb
0905 19:52:24.238324 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.371695 (+133371us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.379710 (+ 8015us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.451657 (+ 71947us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3325261}
I20250905 19:52:24.452195 1185 tablet_replica.cc:331] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.452299 453 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 5b0d4be788f841c2860f5dd65275b623
I20250905 19:52:24.452423 1185 raft_consensus.cc:2241] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.452611 1185 raft_consensus.cc:2270] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.452621 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 4c76077e4adf45a7adc736b0e69032eb (table default.Table_48447ad862474825a10962298c97650d [id=d467b5191bf744858924c0ecbbf2e53c]) successfully deleted
I20250905 19:52:24.452768 453 catalog_manager.cc:6652] ReplaceTablet: tablet 5b0d4be788f841c2860f5dd65275b623 deleted and replaced by tablet 37311372c03644b0a99ce92b5d42bfd1
W20250905 19:52:24.452809 711 leader_election.cc:343] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 7edcf57f7753449d8a5fcdb6f4f1dc21
I20250905 19:52:24.453065 453 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_4e6468a3541a49c6841da695333625ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:24.453349 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74a7e7d9874e4d26bda3ab24a6e24adc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.453547 453 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_4e6468a3541a49c6841da695333625ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.453769 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.457404 1185 ts_tablet_manager.cc:1905] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:24.457480 848 leader_election.cc:343] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.457710 848 leader_election.cc:304] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.458302 1584 raft_consensus.cc:2747] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.459794 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5aff2acc3f8245e1bca41f971f8e74ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.461517 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3602094b96c14e66b997e8035fc53c40" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.464738 450 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_a3f620901eff44dea30bf9e4de3c5552"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:24.465071 848 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:24.465265 450 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a3f620901eff44dea30bf9e4de3c5552 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.465231 848 leader_election.cc:304] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.465728 1584 raft_consensus.cc:2747] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.466487 448 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ba161282480e4736889559a2e37b195f
I20250905 19:52:24.466957 448 catalog_manager.cc:6652] ReplaceTablet: tablet ba161282480e4736889559a2e37b195f deleted and replaced by tablet 8faf429d811347a88880e8198daa59cc
I20250905 19:52:24.467303 448 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_bc301cb31d184494ad445294416d0a20"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:24.467818 448 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_bc301cb31d184494ad445294416d0a20 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:24.469005 452 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 3fe2b75a3e59437c885abc18614f1da2 consensus state has no leader
I20250905 19:52:24.469421 452 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_6016725fc2e44e0381a7cc4e06bcac86"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:24.469679 889 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:51212
W20250905 19:52:24.469911 452 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6016725fc2e44e0381a7cc4e06bcac86 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.465265 455 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 8503f6918ffb44efa36214633c93e31d
I20250905 19:52:24.471421 455 catalog_manager.cc:6652] ReplaceTablet: tablet 8503f6918ffb44efa36214633c93e31d deleted and replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6
I20250905 19:52:24.471690 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.471786 455 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_3a4d5c41d6f44c128dbb1ef09d643d84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:24.472259 455 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.482507 878 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.490670 446 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac] alter complete (version 2)
I20250905 19:52:24.492398 449 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:52586:
name: "default.Table_24293fcc0f3244c4a016cbb7e189a24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:24.493095 449 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24293fcc0f3244c4a016cbb7e189a24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:24.497071 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.497373 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.499040 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.499353 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.499579 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.499778 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:51212
I20250905 19:52:24.500655 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.506944 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu pid 424
I20250905 19:52:24.508810 485 generic_service.cc:196] Checking for leaks (request via RPC)
I20250905 19:52:24.509832 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 180437d1534946619384307ddcb7c937. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.511101 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.541208 1185 ts_tablet_manager.cc:1918] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.541589 1185 log.cc:1199] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/0edac849e8834894be2d4ff323edd13d
I20250905 19:52:24.542186 1185 ts_tablet_manager.cc:1939] T 0edac849e8834894be2d4ff323edd13d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.542189 751 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.542577 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.514451 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b1f46261822442e0a961d2d63bdb35c8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.543510 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 313) took 3630 ms. Trace:
I20250905 19:52:24.543733 1185 rpcz_store.cc:276] 0905 19:52:20.913141 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.913499 (+ 358us) service_pool.cc:225] Handling call
0905 19:52:24.452109 (+3538610us) ts_tablet_manager.cc:1145] Deleting tablet 0edac849e8834894be2d4ff323edd13d
0905 19:52:24.452119 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.472394 (+ 20275us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.539085 (+ 66691us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.543469 (+ 4384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3538339}
I20250905 19:52:24.544160 1185 tablet_replica.cc:331] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.544375 1185 raft_consensus.cc:2241] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.544535 1185 raft_consensus.cc:2270] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.545091 1185 ts_tablet_manager.cc:1905] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.548707 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.549085 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.549448 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 7c5807c1132747e6af79d93e7496d500 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:48712
I20250905 19:52:24.549710 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e63b27ff4e434d76a8280f90c22f4082. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.552301 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f34f0b29d6104cac8ab35309fd1edae8. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.554796 1185 ts_tablet_manager.cc:1918] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.556505 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 74eb2632a6414e52b9ad67f53c182679. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.560230 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dc5428f20ca14fdf88a3a52f7b4d1e19. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.560884 1185 log.cc:1199] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/3602094b96c14e66b997e8035fc53c40
I20250905 19:52:24.561256 1185 ts_tablet_manager.cc:1939] T 3602094b96c14e66b997e8035fc53c40 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.562070 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 314) took 3648 ms. Trace:
I20250905 19:52:24.562199 1185 rpcz_store.cc:276] 0905 19:52:20.913697 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.913806 (+ 109us) service_pool.cc:225] Handling call
0905 19:52:24.544079 (+3630273us) ts_tablet_manager.cc:1145] Deleting tablet 3602094b96c14e66b997e8035fc53c40
0905 19:52:24.544090 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.551653 (+ 7563us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.554686 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.562013 (+ 7327us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3630108}
I20250905 19:52:24.563620 1185 tablet_replica.cc:331] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.563928 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5cbd471695e748c9831f9b0f6795aea7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.563994 1185 raft_consensus.cc:2241] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.564244 1185 raft_consensus.cc:2270] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.564957 1185 ts_tablet_manager.cc:1905] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.567725 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a860ab3f5ce54bd2b75bb71730a3c6e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.572161 746 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b4f9e12781a46f19b563746a70b59e4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.516364 1170 tablet_bootstrap.cc:492] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:24.573820 1185 ts_tablet_manager.cc:1918] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.574031 1185 log.cc:1199] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:24.574505 1185 ts_tablet_manager.cc:1939] T d480531ba94f4b0f9fe40e88fe91512e P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.575003 1170 tablet_bootstrap.cc:654] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.575558 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 315) took 3661 ms. Trace:
I20250905 19:52:24.575773 1185 rpcz_store.cc:276] 0905 19:52:20.914151 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.914199 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:24.563494 (+3649295us) ts_tablet_manager.cc:1145] Deleting tablet d480531ba94f4b0f9fe40e88fe91512e
0905 19:52:24.563510 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.569525 (+ 6015us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.573653 (+ 4128us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.575509 (+ 1856us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3649081}
I20250905 19:52:24.576432 1185 tablet_replica.cc:331] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.576700 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c77dd068ea5c495ca8aa0e7a6823d792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.576756 1185 raft_consensus.cc:2241] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:24.577164 1185 pending_rounds.cc:70] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:24.577286 1185 pending_rounds.cc:77] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089560167911424 op_type: NO_OP noop_request { }
I20250905 19:52:24.577386 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c870e689af254fb0acb15330577e5c42. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.577489 1185 raft_consensus.cc:2887] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:24.577656 1185 raft_consensus.cc:2270] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.518921 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:48712
W20250905 19:52:24.513201 711 leader_election.cc:343] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 76a5c85dbda140d882a5a63a13aaabd7
W20250905 19:52:24.578621 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 97415.9us
I20250905 19:52:24.514863 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.515719 1186 tablet_replica.cc:331] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:24.581291 1186 raft_consensus.cc:2241] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.581529 1186 raft_consensus.cc:2270] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.582160 1186 ts_tablet_manager.cc:1905] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:24.582732 712 leader_election.cc:343] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 76a5c85dbda140d882a5a63a13aaabd7
I20250905 19:52:24.583225 712 leader_election.cc:304] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:24.584128 1849 raft_consensus.cc:2747] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:24.584265 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 72043.2us
W20250905 19:52:24.584595 712 leader_election.cc:343] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 7edcf57f7753449d8a5fcdb6f4f1dc21
I20250905 19:52:24.584770 712 leader_election.cc:304] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:24.586414 1849 raft_consensus.cc:2747] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.578361 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6f29ac5e68d4d8da8f3496fefe695ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.589155 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a551e641a19f4d3f924fb176c4a25ca7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.589722 750 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e6ac13b19fb6427a9ef35a5160033a54. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.590251 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet af10abf0a5b34b3bbfea20e5c2005006. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:24.609537 1185 ts_tablet_manager.cc:1905] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.622092 1171 tablet_bootstrap.cc:492] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:24.622380 1171 ts_tablet_manager.cc:1397] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.203s user 0.004s sys 0.000s
W20250905 19:52:24.622562 984 consensus_peers.cc:489] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:24.622763 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:24.625010 1171 raft_consensus.cc:357] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.625413 1171 raft_consensus.cc:383] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.625546 1171 raft_consensus.cc:738] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.625937 1171 consensus_queue.cc:260] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.626545 1171 ts_tablet_manager.cc:1428] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:24.629246 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:24.629534 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:44176
W20250905 19:52:24.630380 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 141561us
W20250905 19:52:24.633576 984 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.633724 984 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:24.635358 984 leader_election.cc:343] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.635484 984 leader_election.cc:304] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.635809 1809 raft_consensus.cc:2747] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.637733 1809 raft_consensus.cc:2747] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.642047 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:24.643451 1809 raft_consensus.cc:491] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.649322 1809 raft_consensus.cc:513] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.649986 1809 leader_election.cc:290] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.655213 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:44176
I20250905 19:52:24.664063 1185 ts_tablet_manager.cc:1918] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:24.664325 1185 log.cc:1199] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/36f966bf512d4cbe9ae406386db31533
I20250905 19:52:24.666610 1185 ts_tablet_manager.cc:1939] T 36f966bf512d4cbe9ae406386db31533 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.672809 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.674631 984 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.679155 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:24.680012 1171 tablet_bootstrap.cc:492] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:24.680452 986 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.681149 986 leader_election.cc:304] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.681625 1853 raft_consensus.cc:2747] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.682158 1809 consensus_queue.cc:1035] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 19:52:24.682523 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 316) took 3761 ms. Trace:
I20250905 19:52:24.683279 1184 ts_tablet_manager.cc:1918] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.683760 1184 log.cc:1199] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:24.684250 1184 ts_tablet_manager.cc:1939] T 78076f6534814939ae8c2db9a4bd1879 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:24.685653 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 316) took 3722 ms. Trace:
I20250905 19:52:24.686944 1184 rpcz_store.cc:276] 0905 19:52:20.962651 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.963374 (+ 723us) service_pool.cc:225] Handling call
0905 19:52:24.381737 (+3418363us) ts_tablet_manager.cc:1145] Deleting tablet 78076f6534814939ae8c2db9a4bd1879
0905 19:52:24.381749 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.431308 (+ 49559us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.683156 (+251848us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.685555 (+ 2399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3418170}
I20250905 19:52:24.687412 1184 tablet_replica.cc:331] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:24.695048 1184 raft_consensus.cc:2241] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.695412 1184 raft_consensus.cc:2270] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.695499 1185 rpcz_store.cc:276] 0905 19:52:20.921063 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.921120 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:24.576344 (+3655224us) ts_tablet_manager.cc:1145] Deleting tablet 36f966bf512d4cbe9ae406386db31533
0905 19:52:24.576359 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.643089 (+ 66730us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.663897 (+ 20808us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.682481 (+ 18584us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3654966}
I20250905 19:52:24.695142 1169 tablet_bootstrap.cc:492] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:24.696146 1184 ts_tablet_manager.cc:1905] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.696286 1169 ts_tablet_manager.cc:1397] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.336s user 0.040s sys 0.107s
I20250905 19:52:24.697975 1809 raft_consensus.cc:491] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.698124 1809 raft_consensus.cc:513] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.698859 1809 leader_election.cc:290] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.699923 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:24.700718 986 leader_election.cc:343] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.701084 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.701211 1185 tablet_replica.cc:331] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.701427 1185 raft_consensus.cc:2241] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:24.702912 1185 raft_consensus.cc:2270] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:24.704144 984 leader_election.cc:343] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.704296 984 leader_election.cc:304] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.704741 1809 raft_consensus.cc:2747] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.705229 1169 raft_consensus.cc:357] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.705421 1184 ts_tablet_manager.cc:1918] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.717413 1184 log.cc:1199] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:24.718097 1184 ts_tablet_manager.cc:1939] T b0702063ec1c4f6fba58e6d4f3912212 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:24.719159 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 321) took 3755 ms. Trace:
I20250905 19:52:24.705859 1169 raft_consensus.cc:383] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.724448 1169 raft_consensus.cc:738] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.724834 1169 consensus_queue.cc:260] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.728216 1169 ts_tablet_manager.cc:1428] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.032s user 0.001s sys 0.001s
I20250905 19:52:24.729357 1169 tablet_bootstrap.cc:492] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.731590 1169 tablet_bootstrap.cc:654] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.731762 1184 rpcz_store.cc:276] 0905 19:52:20.963720 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.963761 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:24.687327 (+3723566us) ts_tablet_manager.cc:1145] Deleting tablet b0702063ec1c4f6fba58e6d4f3912212
0905 19:52:24.687337 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.701583 (+ 14246us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.705284 (+ 3701us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.719105 (+ 13821us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3723396}
I20250905 19:52:24.736641 1184 tablet_replica.cc:331] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:24.737002 1184 raft_consensus.cc:2241] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.737257 1184 raft_consensus.cc:2270] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.738020 1184 ts_tablet_manager.cc:1905] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.742868 1186 ts_tablet_manager.cc:1918] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.743139 1186 log.cc:1199] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/77cbdda0e2d641bd8092f00925bb9a9c
I20250905 19:52:24.744189 1186 ts_tablet_manager.cc:1939] T 77cbdda0e2d641bd8092f00925bb9a9c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:24.745339 1186 tablet_replica.cc:331] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:24.745651 1186 raft_consensus.cc:2241] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.745931 1186 raft_consensus.cc:2270] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.746610 1186 ts_tablet_manager.cc:1905] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.754048 1170 tablet_bootstrap.cc:492] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:24.754226 1170 ts_tablet_manager.cc:1397] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.243s user 0.007s sys 0.016s
I20250905 19:52:24.754983 1170 raft_consensus.cc:357] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.755223 1170 raft_consensus.cc:383] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.755318 1170 raft_consensus.cc:738] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.755554 1170 consensus_queue.cc:260] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.756104 1170 ts_tablet_manager.cc:1428] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:24.756407 1170 tablet_bootstrap.cc:492] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:24.757758 1169 tablet_bootstrap.cc:492] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:24.757952 1169 ts_tablet_manager.cc:1397] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250905 19:52:24.758915 1169 raft_consensus.cc:357] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.759085 1169 raft_consensus.cc:383] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.759778 1169 raft_consensus.cc:738] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.760172 1169 consensus_queue.cc:260] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.760564 1170 tablet_bootstrap.cc:654] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.762645 1169 ts_tablet_manager.cc:1428] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 19:52:24.762948 1169 tablet_bootstrap.cc:492] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.769260 1169 tablet_bootstrap.cc:654] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.773794 1170 tablet_bootstrap.cc:492] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:24.774125 1170 ts_tablet_manager.cc:1397] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.008s sys 0.000s
I20250905 19:52:24.775173 1170 raft_consensus.cc:357] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.775465 1170 raft_consensus.cc:383] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.775611 1170 raft_consensus.cc:738] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.776001 1170 consensus_queue.cc:260] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.776219 1809 raft_consensus.cc:491] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.776386 1809 raft_consensus.cc:513] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.777029 1809 leader_election.cc:290] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.777781 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:24.778471 986 leader_election.cc:343] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.778765 1170 ts_tablet_manager.cc:1428] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:24.779163 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.780105 1170 tablet_bootstrap.cc:492] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:24.781126 984 leader_election.cc:343] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.781194 1184 ts_tablet_manager.cc:1918] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.781404 984 leader_election.cc:304] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.781864 1809 raft_consensus.cc:2747] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.788278 1185 ts_tablet_manager.cc:1905] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.791270 1169 tablet_bootstrap.cc:492] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:24.791527 1169 ts_tablet_manager.cc:1397] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20250905 19:52:24.792155 1672 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.792528 1169 raft_consensus.cc:357] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.792809 1169 raft_consensus.cc:383] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.792924 1169 raft_consensus.cc:738] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.792826 1672 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.793268 1169 consensus_queue.cc:260] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.793608 1672 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.794381 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.794991 848 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.795986 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:24.796231 1047 raft_consensus.cc:2466] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:24.797822 849 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.798055 1658 raft_consensus.cc:491] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.798195 1658 raft_consensus.cc:513] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.798417 1672 raft_consensus.cc:2802] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:24.798514 1672 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:24.798595 1672 raft_consensus.cc:3058] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:24.798956 1658 leader_election.cc:290] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:24.800035 849 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:24.801321 1672 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.802809 1672 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.803328 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:24.803608 1047 raft_consensus.cc:3058] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:24.806104 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "68f2ccd8a7a64b89b44f2d6767ec3466" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:24.806759 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2d84653c9e7a4857a7580fa85f8e4732" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:24.807144 1047 raft_consensus.cc:2466] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:24.807219 848 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.781517 1184 log.cc:1199] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:24.807355 848 leader_election.cc:304] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.807627 849 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:24.807847 1184 ts_tablet_manager.cc:1939] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:24.807834 848 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.808645 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 322) took 3844 ms. Trace:
I20250905 19:52:24.808745 1184 rpcz_store.cc:276] 0905 19:52:20.964079 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.964121 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:24.736551 (+3772430us) ts_tablet_manager.cc:1145] Deleting tablet 4d2ed1bc09c94526bc9662f2d4d21f8f
0905 19:52:24.736565 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.756818 (+ 20253us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.781023 (+ 24205us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.808580 (+ 27557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3772237}
I20250905 19:52:24.809180 1184 tablet_replica.cc:331] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:24.809695 1584 raft_consensus.cc:2747] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.809899 1184 raft_consensus.cc:2241] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.809939 1584 raft_consensus.cc:2802] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:24.810089 1584 raft_consensus.cc:695] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:24.810114 1184 raft_consensus.cc:2270] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.810360 1584 consensus_queue.cc:237] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.810648 1184 ts_tablet_manager.cc:1905] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.811072 1169 ts_tablet_manager.cc:1428] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250905 19:52:24.811471 1798 raft_consensus.cc:491] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.811615 1798 raft_consensus.cc:513] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.812309 1798 leader_election.cc:290] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.813505 1169 tablet_bootstrap.cc:492] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.699028 1171 tablet_bootstrap.cc:654] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.815879 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:24.818657 986 leader_election.cc:343] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.820848 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.821512 984 leader_election.cc:343] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.821671 984 leader_election.cc:304] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.821843 1169 tablet_bootstrap.cc:654] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.822165 1798 raft_consensus.cc:2747] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.824318 1658 raft_consensus.cc:491] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.824486 1658 raft_consensus.cc:513] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.825206 1658 leader_election.cc:290] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.826047 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe94d1254d04949aa25346d8e323c3b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:24.826260 1047 raft_consensus.cc:2466] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:24.826813 849 leader_election.cc:304] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:24.827682 1658 raft_consensus.cc:2802] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:24.827796 1658 raft_consensus.cc:491] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:24.827890 1658 raft_consensus.cc:3058] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:24.827952 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe94d1254d04949aa25346d8e323c3b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.830111 848 leader_election.cc:343] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.831185 1658 raft_consensus.cc:513] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.836648 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe94d1254d04949aa25346d8e323c3b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:24.836889 1047 raft_consensus.cc:3058] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:24.837801 1170 tablet_bootstrap.cc:654] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.845642 1169 tablet_bootstrap.cc:492] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:24.845809 1169 ts_tablet_manager.cc:1397] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.005s
I20250905 19:52:24.847158 1169 raft_consensus.cc:357] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.847357 1169 raft_consensus.cc:383] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.847446 1169 raft_consensus.cc:738] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.847436 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4fe94d1254d04949aa25346d8e323c3b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:24.847668 1169 consensus_queue.cc:260] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:24.852747 848 leader_election.cc:343] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.854877 1171 tablet_bootstrap.cc:492] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:24.855039 1171 ts_tablet_manager.cc:1397] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.175s user 0.002s sys 0.002s
I20250905 19:52:24.855034 1169 ts_tablet_manager.cc:1428] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20250905 19:52:24.855865 1171 raft_consensus.cc:357] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.856081 1171 raft_consensus.cc:383] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.856161 1171 raft_consensus.cc:738] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.856510 1171 consensus_queue.cc:260] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.861620 1047 raft_consensus.cc:2466] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:24.862246 849 leader_election.cc:304] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.862591 1658 leader_election.cc:290] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.866696 1186 ts_tablet_manager.cc:1918] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.866847 1186 log.cc:1199] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e6f837fb7293474b8ad357f52e42e42e
I20250905 19:52:24.867201 1186 ts_tablet_manager.cc:1939] T e6f837fb7293474b8ad357f52e42e42e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:24.868853 1170 tablet_bootstrap.cc:492] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:24.869102 1170 ts_tablet_manager.cc:1397] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.089s user 0.004s sys 0.000s
I20250905 19:52:24.870296 1186 tablet_replica.cc:331] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:24.870669 1170 raft_consensus.cc:357] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.870939 1170 raft_consensus.cc:383] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.871085 1170 raft_consensus.cc:738] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.871392 1170 consensus_queue.cc:260] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.874186 1672 raft_consensus.cc:2802] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:24.874372 1672 raft_consensus.cc:695] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:24.874642 1672 consensus_queue.cc:237] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.879108 1169 tablet_bootstrap.cc:492] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.882237 1170 ts_tablet_manager.cc:1428] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:24.884244 1169 tablet_bootstrap.cc:654] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.886011 1809 raft_consensus.cc:491] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.886247 1809 raft_consensus.cc:513] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.887018 1809 leader_election.cc:290] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.887946 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff2f289c293f41379e4af470d747860c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:24.888578 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff2f289c293f41379e4af470d747860c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.889132 986 leader_election.cc:343] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:24.889406 984 leader_election.cc:343] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.889547 984 leader_election.cc:304] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.891556 1809 raft_consensus.cc:2747] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.896581 1170 tablet_bootstrap.cc:492] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:24.899367 1913 tablet.cc:1722] T 2466a1bafc8347e483ca5e28d552bfc9 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:24.900843 1186 raft_consensus.cc:2241] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.901152 1915 tablet.cc:1722] T 2466a1bafc8347e483ca5e28d552bfc9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:24.901329 1186 raft_consensus.cc:2270] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.898801 1170 tablet_bootstrap.cc:654] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.901968 1186 ts_tablet_manager.cc:1905] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.904119 1849 raft_consensus.cc:491] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.904498 1171 ts_tablet_manager.cc:1428] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.049s user 0.002s sys 0.000s
I20250905 19:52:24.904856 1171 tablet_bootstrap.cc:492] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:24.906579 1171 tablet_bootstrap.cc:654] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.910243 1185 ts_tablet_manager.cc:1918] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:24.910403 1185 log.cc:1199] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/2c884954e5dc4445a12252953593d0bc
I20250905 19:52:24.910837 1185 ts_tablet_manager.cc:1939] T 2c884954e5dc4445a12252953593d0bc P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.912775 1169 tablet_bootstrap.cc:492] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:24.913012 1169 ts_tablet_manager.cc:1397] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.048s user 0.002s sys 0.005s
I20250905 19:52:24.914351 1169 raft_consensus.cc:357] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.914719 1169 raft_consensus.cc:383] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.914845 1169 raft_consensus.cc:738] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.915215 1169 consensus_queue.cc:260] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.919083 1184 ts_tablet_manager.cc:1918] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.919230 1184 log.cc:1199] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:24.919612 1184 ts_tablet_manager.cc:1939] T b13d282e1dc54d1faa2882c9b39d17cb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:24.920054 1849 raft_consensus.cc:513] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.920902 1849 leader_election.cc:290] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:24.921721 1809 raft_consensus.cc:491] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.921872 1809 raft_consensus.cc:513] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.922472 1809 leader_election.cc:290] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.922844 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:24.923202 712 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 19e4edddafcd4d22a0e0ae66b0b953e0
W20250905 19:52:24.923504 986 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.924680 1169 ts_tablet_manager.cc:1428] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250905 19:52:24.926841 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.927102 711 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:24.927625 711 leader_election.cc:304] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:24.928220 1171 tablet_bootstrap.cc:492] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:24.928380 1171 ts_tablet_manager.cc:1397] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250905 19:52:24.929090 1171 raft_consensus.cc:357] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.929292 1171 raft_consensus.cc:383] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.929363 1171 raft_consensus.cc:738] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.929592 1171 consensus_queue.cc:260] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.929852 1849 raft_consensus.cc:2747] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:24.927392 984 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.931452 984 leader_election.cc:304] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.932087 1809 raft_consensus.cc:2747] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.933485 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 317) took 4005 ms. Trace:
I20250905 19:52:24.933595 1185 rpcz_store.cc:276] 0905 19:52:20.927736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.927793 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:24.701144 (+3773351us) ts_tablet_manager.cc:1145] Deleting tablet 2c884954e5dc4445a12252953593d0bc
0905 19:52:24.701154 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.791825 (+ 90671us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.910106 (+118281us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.933425 (+ 23319us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3773093}
I20250905 19:52:24.937513 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 319) took 3988 ms. Trace:
I20250905 19:52:24.937619 1185 rpcz_store.cc:276] 0905 19:52:20.949090 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.950372 (+ 1282us) service_pool.cc:225] Handling call
0905 19:52:24.937326 (+3986954us) ts_tablet_manager.cc:1145] Deleting tablet b13d282e1dc54d1faa2882c9b39d17cb
0905 19:52:24.937339 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.937458 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3986733}
I20250905 19:52:24.938156 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 329) took 3915 ms. Trace:
I20250905 19:52:24.938249 1185 rpcz_store.cc:276] 0905 19:52:21.023145 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.023200 (+ 55us) service_pool.cc:225] Handling call
0905 19:52:24.938004 (+3914804us) ts_tablet_manager.cc:1145] Deleting tablet f48f9f789b6e46ec871a0fd007770dfc
0905 19:52:24.938013 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.938109 (+ 96us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3914550}
I20250905 19:52:24.938915 1171 ts_tablet_manager.cc:1428] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250905 19:52:24.939227 1171 tablet_bootstrap.cc:492] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:24.940994 1171 tablet_bootstrap.cc:654] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.942019 1185 tablet_replica.cc:331] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.942237 1185 raft_consensus.cc:2241] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:24.942503 1185 pending_rounds.cc:70] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:24.942592 1185 pending_rounds.cc:77] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089565945544704 op_type: NO_OP noop_request { }
I20250905 19:52:24.942724 1185 raft_consensus.cc:2887] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:24.942832 1185 raft_consensus.cc:2270] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.950951 1186 ts_tablet_manager.cc:1918] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:24.952611 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 323) took 3988 ms. Trace:
I20250905 19:52:24.952803 1184 rpcz_store.cc:276] 0905 19:52:20.964452 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.964495 (+ 43us) service_pool.cc:225] Handling call
0905 19:52:24.809095 (+3844600us) ts_tablet_manager.cc:1145] Deleting tablet b13d282e1dc54d1faa2882c9b39d17cb
0905 19:52:24.809104 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.915549 (+106445us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.918938 (+ 3389us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.952559 (+ 33621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3844422}
I20250905 19:52:24.953758 1169 tablet_bootstrap.cc:492] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.955946 1169 tablet_bootstrap.cc:654] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.956789 1184 tablet_replica.cc:331] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:24.957027 1184 raft_consensus.cc:2241] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.957245 1184 raft_consensus.cc:2270] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.957360 1185 ts_tablet_manager.cc:1905] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.957823 1184 ts_tablet_manager.cc:1905] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.956776 1186 log.cc:1199] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/76a5c85dbda140d882a5a63a13aaabd7
I20250905 19:52:24.962953 1186 ts_tablet_manager.cc:1939] T 76a5c85dbda140d882a5a63a13aaabd7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:24.964236 1186 tablet_replica.cc:331] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:24.964537 1186 raft_consensus.cc:2241] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.968214 1186 raft_consensus.cc:2270] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.968945 1186 ts_tablet_manager.cc:1905] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.970971 1184 ts_tablet_manager.cc:1918] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:24.971131 1184 log.cc:1199] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/2c884954e5dc4445a12252953593d0bc
I20250905 19:52:24.971539 1184 ts_tablet_manager.cc:1939] T 2c884954e5dc4445a12252953593d0bc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:24.972155 1170 tablet_bootstrap.cc:492] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:24.972373 1170 ts_tablet_manager.cc:1397] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.076s user 0.004s sys 0.006s
I20250905 19:52:24.972400 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 315) took 4009 ms. Trace:
I20250905 19:52:24.972499 1184 rpcz_store.cc:276] 0905 19:52:20.962538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.965501 (+ 2963us) service_pool.cc:225] Handling call
0905 19:52:24.953253 (+3987752us) ts_tablet_manager.cc:1145] Deleting tablet 2c884954e5dc4445a12252953593d0bc
0905 19:52:24.955992 (+ 2739us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.961036 (+ 5044us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.970842 (+ 9806us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.972341 (+ 1499us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":3840,"tablet-delete.queue_time_us":3987581}
I20250905 19:52:24.973574 1170 raft_consensus.cc:357] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.973937 1170 raft_consensus.cc:383] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.974093 1170 raft_consensus.cc:738] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.974465 1170 consensus_queue.cc:260] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.975796 1169 tablet_bootstrap.cc:492] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:24.975948 1169 ts_tablet_manager.cc:1397] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.005s
I20250905 19:52:24.976806 1169 raft_consensus.cc:357] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.977003 1169 raft_consensus.cc:383] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.977100 1169 raft_consensus.cc:738] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.977385 1169 consensus_queue.cc:260] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.978096 1170 ts_tablet_manager.cc:1428] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:24.979463 1170 tablet_bootstrap.cc:492] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:24.980330 1185 ts_tablet_manager.cc:1918] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:24.980476 1185 log.cc:1199] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:24.980924 1185 ts_tablet_manager.cc:1939] T 2d84653c9e7a4857a7580fa85f8e4732 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:24.981782 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 328) took 3960 ms. Trace:
I20250905 19:52:24.981887 1185 rpcz_store.cc:276] 0905 19:52:21.021533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.021593 (+ 60us) service_pool.cc:225] Handling call
0905 19:52:24.941936 (+3920343us) ts_tablet_manager.cc:1145] Deleting tablet 2d84653c9e7a4857a7580fa85f8e4732
0905 19:52:24.941944 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.961027 (+ 19083us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.980178 (+ 19151us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:24.981726 (+ 1548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3889876}
I20250905 19:52:24.982232 1185 tablet_replica.cc:331] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:24.982335 1184 tablet_replica.cc:331] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:24.982438 1185 raft_consensus.cc:2241] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:24.982560 1184 raft_consensus.cc:2241] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:24.982760 1184 raft_consensus.cc:2270] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.982807 1185 raft_consensus.cc:2270] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:24.983322 1184 ts_tablet_manager.cc:1905] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.983373 1185 ts_tablet_manager.cc:1905] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:24.985092 1809 raft_consensus.cc:491] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:24.985509 1809 raft_consensus.cc:513] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.986141 1809 leader_election.cc:290] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:24.987074 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:24.987116 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:24.987622 984 leader_election.cc:343] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:24.987838 986 leader_election.cc:343] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:24.987957 986 leader_election.cc:304] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:24.988464 1809 raft_consensus.cc:2747] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:24.988631 1171 tablet_bootstrap.cc:492] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:24.988768 1171 ts_tablet_manager.cc:1397] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.050s user 0.003s sys 0.009s
I20250905 19:52:24.989068 1169 ts_tablet_manager.cc:1428] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.013s user 0.000s sys 0.002s
I20250905 19:52:24.989734 1171 raft_consensus.cc:357] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.989907 1171 raft_consensus.cc:383] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:24.989984 1171 raft_consensus.cc:738] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:24.990226 1171 consensus_queue.cc:260] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:24.992717 1171 ts_tablet_manager.cc:1428] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250905 19:52:24.993060 1171 tablet_bootstrap.cc:492] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:24.994097 1169 tablet_bootstrap.cc:492] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:24.994966 1171 tablet_bootstrap.cc:654] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:24.996513 1169 tablet_bootstrap.cc:654] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.002454 1170 tablet_bootstrap.cc:654] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.002470 1184 ts_tablet_manager.cc:1918] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.006311 1184 log.cc:1199] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:25.008800 1185 ts_tablet_manager.cc:1918] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:25.008991 1185 log.cc:1199] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/fbab0209a48d4ad8b2f2dec0562e9598
I20250905 19:52:25.009092 1184 ts_tablet_manager.cc:1939] T 4fe94d1254d04949aa25346d8e323c3b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.009552 1185 ts_tablet_manager.cc:1939] T fbab0209a48d4ad8b2f2dec0562e9598 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.010609 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 320) took 4047 ms. Trace:
I20250905 19:52:25.010753 1184 rpcz_store.cc:276] 0905 19:52:20.963595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.966696 (+ 3101us) service_pool.cc:225] Handling call
0905 19:52:24.975157 (+4008461us) ts_tablet_manager.cc:1145] Deleting tablet 4fe94d1254d04949aa25346d8e323c3b
0905 19:52:24.981032 (+ 5875us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.999285 (+ 18253us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.002330 (+ 3045us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.010570 (+ 8240us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4008273}
I20250905 19:52:25.010878 1186 ts_tablet_manager.cc:1918] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.010995 1186 log.cc:1199] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/57f851a6eecc4a41806bdaa19cc5afab
I20250905 19:52:25.011080 1171 tablet_bootstrap.cc:492] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.011288 1171 ts_tablet_manager.cc:1397] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.002s
I20250905 19:52:25.012009 1171 raft_consensus.cc:357] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.012238 1171 raft_consensus.cc:383] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.012275 1184 tablet_replica.cc:331] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.012305 1171 raft_consensus.cc:738] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.012550 1171 consensus_queue.cc:260] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.013041 1171 ts_tablet_manager.cc:1428] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:25.017021 1171 tablet_bootstrap.cc:492] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.018754 1184 raft_consensus.cc:2241] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.018940 1184 pending_rounds.cc:70] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:25.019024 1184 pending_rounds.cc:77] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089560167911424 op_type: NO_OP noop_request { }
I20250905 19:52:25.019130 1184 raft_consensus.cc:2887] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:25.019254 1184 raft_consensus.cc:2270] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.021281 1184 ts_tablet_manager.cc:1905] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.016067 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 331) took 3950 ms. Trace:
I20250905 19:52:25.024245 1185 rpcz_store.cc:276] 0905 19:52:21.065824 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.065876 (+ 52us) service_pool.cc:225] Handling call
0905 19:52:24.982156 (+3916280us) ts_tablet_manager.cc:1145] Deleting tablet fbab0209a48d4ad8b2f2dec0562e9598
0905 19:52:24.982165 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.001474 (+ 19309us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.008665 (+ 7191us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.016019 (+ 7354us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3916018}
I20250905 19:52:25.024847 1171 tablet_bootstrap.cc:654] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.025108 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 339) took 3698 ms. Trace:
I20250905 19:52:25.025295 1185 rpcz_store.cc:276] 0905 19:52:21.326327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.326442 (+ 115us) service_pool.cc:225] Handling call
0905 19:52:25.024867 (+3698425us) ts_tablet_manager.cc:1145] Deleting tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
0905 19:52:25.024879 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.025065 (+ 186us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3691096}
I20250905 19:52:25.025915 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 341) took 3573 ms. Trace:
I20250905 19:52:25.026111 1185 rpcz_store.cc:276] 0905 19:52:21.452753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.452809 (+ 56us) service_pool.cc:225] Handling call
0905 19:52:25.025757 (+3572948us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:25.025767 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.025873 (+ 106us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3572722}
I20250905 19:52:25.026635 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 344) took 3555 ms. Trace:
I20250905 19:52:25.026799 1185 rpcz_store.cc:276] 0905 19:52:21.471023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.479289 (+ 8266us) service_pool.cc:225] Handling call
0905 19:52:25.026516 (+3547227us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:25.026524 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.026603 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3538242}
I20250905 19:52:25.027311 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 353) took 3342 ms. Trace:
I20250905 19:52:25.027526 1185 rpcz_store.cc:276] 0905 19:52:21.685122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.685258 (+ 136us) service_pool.cc:225] Handling call
0905 19:52:25.027194 (+3341936us) ts_tablet_manager.cc:1145] Deleting tablet 8edf4b54974c427a855eee5792d75292
0905 19:52:25.027202 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.027274 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3323177}
I20250905 19:52:25.025205 1170 tablet_bootstrap.cc:492] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.028084 1170 ts_tablet_manager.cc:1397] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.050s user 0.005s sys 0.000s
I20250905 19:52:25.028188 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 346) took 3534 ms. Trace:
I20250905 19:52:25.029238 1185 rpcz_store.cc:276] 0905 19:52:21.493561 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.587986 (+ 94425us) service_pool.cc:225] Handling call
0905 19:52:25.028037 (+3440051us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:25.028049 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.028148 (+ 99us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3313051}
I20250905 19:52:25.029552 1170 raft_consensus.cc:357] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.029842 1185 tablet_replica.cc:331] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.030154 1185 raft_consensus.cc:2241] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:25.030386 1170 raft_consensus.cc:383] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.030592 1185 raft_consensus.cc:2270] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.030869 1170 raft_consensus.cc:738] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.031339 1185 ts_tablet_manager.cc:1905] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.032260 1170 consensus_queue.cc:260] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.036293 1925 raft_consensus.cc:491] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c401c9f3020b4ae5931c566c6bcb1711)
I20250905 19:52:25.036542 1925 raft_consensus.cc:513] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:25.037055 1924 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader c401c9f3020b4ae5931c566c6bcb1711)
I20250905 19:52:25.037391 1924 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:25.038511 1925 leader_election.cc:290] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317)
I20250905 19:52:25.031687 1186 ts_tablet_manager.cc:1939] T 57f851a6eecc4a41806bdaa19cc5afab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:25.045756 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48712 (request call id 410) took 1066 ms. Trace:
I20250905 19:52:25.046026 1169 tablet_bootstrap.cc:492] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.046208 1169 ts_tablet_manager.cc:1397] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.052s user 0.021s sys 0.005s
I20250905 19:52:25.046031 1186 rpcz_store.cc:276] 0905 19:52:23.979648 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.980296 (+ 648us) service_pool.cc:225] Handling call
0905 19:52:24.964149 (+983853us) ts_tablet_manager.cc:1145] Deleting tablet 57f851a6eecc4a41806bdaa19cc5afab
0905 19:52:24.964159 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:24.986836 (+ 22677us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.010738 (+ 23902us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.045703 (+ 34965us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":5470,"tablet-delete.queue_time_us":583574}
I20250905 19:52:25.039254 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 3 candidate_status { last_received { term: 2 index: 593 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" is_pre_election: true
I20250905 19:52:25.047029 1169 raft_consensus.cc:357] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.047257 1169 raft_consensus.cc:383] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.047350 1169 raft_consensus.cc:738] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.047646 1169 consensus_queue.cc:260] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.047925 534 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e38333af29db4bf6af41ee5a25d6701e in term 2.
I20250905 19:52:25.048799 1472 leader_election.cc:304] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91, e38333af29db4bf6af41ee5a25d6701e; no voters:
I20250905 19:52:25.050199 1184 ts_tablet_manager.cc:1918] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:25.050333 1184 log.cc:1199] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/36f966bf512d4cbe9ae406386db31533
I20250905 19:52:25.050750 1184 ts_tablet_manager.cc:1939] T 36f966bf512d4cbe9ae406386db31533 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.051070 1186 tablet_replica.cc:331] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:25.051483 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 314) took 4089 ms. Trace:
I20250905 19:52:25.051611 1184 rpcz_store.cc:276] 0905 19:52:20.962423 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.967145 (+ 4722us) service_pool.cc:225] Handling call
0905 19:52:25.012201 (+4045056us) ts_tablet_manager.cc:1145] Deleting tablet 36f966bf512d4cbe9ae406386db31533
0905 19:52:25.012213 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.046780 (+ 34567us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.050062 (+ 3282us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.051444 (+ 1382us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4044879}
I20250905 19:52:25.051921 1184 tablet_replica.cc:331] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.050011 1170 ts_tablet_manager.cc:1428] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.022s user 0.003s sys 0.000s
I20250905 19:52:25.054795 1186 raft_consensus.cc:2241] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.057354 1924 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:25.057731 1171 tablet_bootstrap.cc:492] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.057883 1171 ts_tablet_manager.cc:1397] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.003s
I20250905 19:52:25.058094 1186 raft_consensus.cc:2270] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.058547 1170 tablet_bootstrap.cc:492] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.058600 1171 raft_consensus.cc:357] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.058776 1171 raft_consensus.cc:383] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.058857 1171 raft_consensus.cc:738] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.058897 1186 ts_tablet_manager.cc:1905] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.059072 1171 consensus_queue.cc:260] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.059381 1925 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Leader pre-election won for term 3
I20250905 19:52:25.059547 1171 ts_tablet_manager.cc:1428] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:25.059870 1171 tablet_bootstrap.cc:492] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.061911 1170 tablet_bootstrap.cc:654] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.063738 1171 tablet_bootstrap.cc:654] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.056588 1169 ts_tablet_manager.cc:1428] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20250905 19:52:25.058436 1809 raft_consensus.cc:491] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.064139 1809 raft_consensus.cc:513] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.064893 1809 leader_election.cc:290] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.065546 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.065886 1169 tablet_bootstrap.cc:492] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.065945 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:25.066296 984 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.061548 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 3 candidate_status { last_received { term: 2 index: 593 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
W20250905 19:52:25.067409 986 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.067538 986 leader_election.cc:304] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.067822 1184 raft_consensus.cc:2241] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.067988 1184 raft_consensus.cc:2270] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.068212 1571 raft_consensus.cc:2466] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c6c0f2b9d8cd457a9c3448034c3d5c91 in term 2.
I20250905 19:52:25.068775 1184 ts_tablet_manager.cc:1905] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.068965 512 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91, e38333af29db4bf6af41ee5a25d6701e; no voters:
I20250905 19:52:25.072801 1925 raft_consensus.cc:491] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Starting leader election (detected failure of leader c401c9f3020b4ae5931c566c6bcb1711)
I20250905 19:52:25.072990 1925 raft_consensus.cc:3058] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:52:25.077836 1171 tablet_bootstrap.cc:492] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.078029 1171 ts_tablet_manager.cc:1397] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.008s
I20250905 19:52:25.078150 1672 raft_consensus.cc:491] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.078328 1672 raft_consensus.cc:513] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.078871 1171 raft_consensus.cc:357] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.078972 1672 leader_election.cc:290] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.079077 1171 raft_consensus.cc:383] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.079146 1171 raft_consensus.cc:738] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.079402 1171 consensus_queue.cc:260] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.080261 849 leader_election.cc:343] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 262bcd1cfc994bb58e999b141b9bc7ea
I20250905 19:52:25.080459 1925 raft_consensus.cc:513] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:25.082624 1672 raft_consensus.cc:491] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.082667 1185 ts_tablet_manager.cc:1918] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:25.082793 1672 raft_consensus.cc:513] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.082911 1185 log.cc:1199] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/2f581403ec924fb0948400df41ed8aab
I20250905 19:52:25.083389 1185 ts_tablet_manager.cc:1939] T 2f581403ec924fb0948400df41ed8aab P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.083487 1672 leader_election.cc:290] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.083829 534 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 3 candidate_status { last_received { term: 2 index: 593 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
I20250905 19:52:25.084055 534 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 2 FOLLOWER]: Advancing to term 3
W20250905 19:52:25.084247 849 leader_election.cc:343] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:25.084322 1171 ts_tablet_manager.cc:1428] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.002s
I20250905 19:52:25.084750 1171 tablet_bootstrap.cc:492] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.085232 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 366) took 2767 ms. Trace:
I20250905 19:52:25.085357 1185 rpcz_store.cc:276] 0905 19:52:22.316356 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.318537 (+ 2181us) service_pool.cc:225] Handling call
0905 19:52:25.029765 (+2711228us) ts_tablet_manager.cc:1145] Deleting tablet 2f581403ec924fb0948400df41ed8aab
0905 19:52:25.029778 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.041648 (+ 11870us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.082515 (+ 40867us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.084144 (+ 1629us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2698525}
I20250905 19:52:25.085984 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 364) took 2776 ms. Trace:
I20250905 19:52:25.086072 1185 rpcz_store.cc:276] 0905 19:52:22.315781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.318570 (+ 2789us) service_pool.cc:225] Handling call
0905 19:52:25.085804 (+2767234us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:25.085816 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.085930 (+ 114us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2656570}
I20250905 19:52:25.086475 1171 tablet_bootstrap.cc:654] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.086828 534 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e38333af29db4bf6af41ee5a25d6701e in term 3.
I20250905 19:52:25.087616 1472 leader_election.cc:304] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91, e38333af29db4bf6af41ee5a25d6701e; no voters:
I20250905 19:52:25.067597 1169 tablet_bootstrap.cc:654] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.092262 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 365) took 2776 ms. Trace:
I20250905 19:52:25.092438 1185 rpcz_store.cc:276] 0905 19:52:22.316215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.318637 (+ 2422us) service_pool.cc:225] Handling call
0905 19:52:25.092103 (+2773466us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:25.092112 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.092209 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2650511}
I20250905 19:52:25.093165 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 385) took 2501 ms. Trace:
I20250905 19:52:25.093374 1185 rpcz_store.cc:276] 0905 19:52:22.591297 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.592087 (+ 790us) service_pool.cc:225] Handling call
0905 19:52:25.092958 (+2500871us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:25.092978 (+ 20us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.093096 (+ 118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2386862}
I20250905 19:52:25.094098 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 387) took 2440 ms. Trace:
I20250905 19:52:25.094269 1185 rpcz_store.cc:276] 0905 19:52:22.653788 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.653860 (+ 72us) service_pool.cc:225] Handling call
0905 19:52:25.093941 (+2440081us) ts_tablet_manager.cc:1145] Deleting tablet 24bcd21f19ba465aa550102e0fccb674
0905 19:52:25.093953 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.094050 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873100}
I20250905 19:52:25.094720 1185 tablet_replica.cc:331] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.095072 1185 raft_consensus.cc:2241] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:25.095078 1924 raft_consensus.cc:2762] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader pre-election decision vote started in defunct term 2: won
I20250905 19:52:25.095661 1185 raft_consensus.cc:2270] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.096222 1184 ts_tablet_manager.cc:1918] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.096380 1184 log.cc:1199] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/42f3c1efb94e446fb0128b6b412821e2
I20250905 19:52:25.096329 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "262bcd1cfc994bb58e999b141b9bc7ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.096711 1184 ts_tablet_manager.cc:1939] T 42f3c1efb94e446fb0128b6b412821e2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.097419 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "302053e1611b4e22a9a2e7b60ae68f27" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.097555 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 317) took 4134 ms. Trace:
I20250905 19:52:25.097640 1184 rpcz_store.cc:276] 0905 19:52:20.962735 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:20.967885 (+ 5150us) service_pool.cc:225] Handling call
0905 19:52:25.051870 (+4083985us) ts_tablet_manager.cc:1145] Deleting tablet 42f3c1efb94e446fb0128b6b412821e2
0905 19:52:25.051879 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.071870 (+ 19991us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.096095 (+ 24225us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.097493 (+ 1398us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4083769}
I20250905 19:52:25.098462 776 raft_consensus.cc:3058] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.103892 1925 leader_election.cc:290] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 election: Requested vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317)
I20250905 19:52:25.104524 1925 raft_consensus.cc:2802] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 3 FOLLOWER]: Leader election won for term 3
I20250905 19:52:25.105307 1925 raft_consensus.cc:695] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [term 3 LEADER]: Becoming Leader. State: Replica: e38333af29db4bf6af41ee5a25d6701e, State: Running, Role: LEADER
I20250905 19:52:25.106035 1925 consensus_queue.cc:237] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 593, Committed index: 593, Last appended: 2.593, Last appended by leader: 593, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:25.108757 1184 tablet_replica.cc:331] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.108986 1184 raft_consensus.cc:2241] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.109225 1184 raft_consensus.cc:2270] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.109442 1186 ts_tablet_manager.cc:1918] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.111562 1930 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38333af29db4bf6af41ee5a25d6701e. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:25.111919 1930 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:25.112365 1184 ts_tablet_manager.cc:1905] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.112577 1169 tablet_bootstrap.cc:492] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.112732 1169 ts_tablet_manager.cc:1397] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.047s user 0.006s sys 0.002s
I20250905 19:52:25.112562 1186 log.cc:1199] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/7edcf57f7753449d8a5fcdb6f4f1dc21
I20250905 19:52:25.113276 1186 ts_tablet_manager.cc:1939] T 7edcf57f7753449d8a5fcdb6f4f1dc21 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:25.097146 848 leader_election.cc:343] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.113646 848 leader_election.cc:304] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.113968 1672 raft_consensus.cc:2747] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.113700 1169 raft_consensus.cc:357] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.114207 848 leader_election.cc:343] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.114393 1169 raft_consensus.cc:383] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.114387 848 leader_election.cc:304] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.114630 1672 raft_consensus.cc:2747] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.114872 1169 raft_consensus.cc:738] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.115217 1169 consensus_queue.cc:260] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.115597 1186 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48712 (request call id 411) took 1135 ms. Trace:
I20250905 19:52:25.115705 1186 rpcz_store.cc:276] 0905 19:52:23.979740 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.980494 (+ 754us) service_pool.cc:225] Handling call
0905 19:52:25.050984 (+1070490us) ts_tablet_manager.cc:1145] Deleting tablet 7edcf57f7753449d8a5fcdb6f4f1dc21
0905 19:52:25.050998 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.106023 (+ 55025us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.109304 (+ 3281us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.115542 (+ 6238us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":670369}
I20250905 19:52:25.116273 1185 ts_tablet_manager.cc:1905] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.116884 1171 tablet_bootstrap.cc:492] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.117038 1171 ts_tablet_manager.cc:1397] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.032s user 0.006s sys 0.003s
I20250905 19:52:25.117867 1171 raft_consensus.cc:357] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.118079 1171 raft_consensus.cc:383] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.118182 1171 raft_consensus.cc:738] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.118453 1171 consensus_queue.cc:260] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.124691 1171 ts_tablet_manager.cc:1428] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 19:52:25.125260 1170 tablet_bootstrap.cc:492] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.125582 1170 ts_tablet_manager.cc:1397] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.004s
I20250905 19:52:25.126708 1170 raft_consensus.cc:357] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.126981 1170 raft_consensus.cc:383] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.127087 1170 raft_consensus.cc:738] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.127429 1170 consensus_queue.cc:260] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.127943 1170 ts_tablet_manager.cc:1428] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:25.128342 1170 tablet_bootstrap.cc:492] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.128446 1934 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:52:25.129587 1171 tablet_bootstrap.cc:492] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.130976 1185 ts_tablet_manager.cc:1918] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:25.131127 1185 log.cc:1199] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/cf2881d228964cd3bef040a06045d8d2
I20250905 19:52:25.131445 1171 tablet_bootstrap.cc:654] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.131621 1185 ts_tablet_manager.cc:1939] T cf2881d228964cd3bef040a06045d8d2 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.134177 1935 tablet.cc:1722] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:25.134986 1186 tablet_replica.cc:331] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:25.135543 1672 raft_consensus.cc:491] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.142650 1672 raft_consensus.cc:513] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.143404 1672 leader_election.cc:290] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.144327 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6109bdf8f4caf8c223a42401b94ab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.144910 848 leader_election.cc:343] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.145470 1186 raft_consensus.cc:2241] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.149230 1171 tablet_bootstrap.cc:492] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.149389 1171 ts_tablet_manager.cc:1397] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.005s
I20250905 19:52:25.149673 1186 raft_consensus.cc:2270] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.150101 1171 raft_consensus.cc:357] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.150254 1186 ts_tablet_manager.cc:1905] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.150288 1171 raft_consensus.cc:383] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.150348 1171 raft_consensus.cc:738] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.150611 1171 consensus_queue.cc:260] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.153281 849 leader_election.cc:343] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:25.153450 849 leader_election.cc:304] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.153999 1672 raft_consensus.cc:2747] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.154422 1171 ts_tablet_manager.cc:1428] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:25.156417 1169 ts_tablet_manager.cc:1428] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.044s user 0.000s sys 0.003s
I20250905 19:52:25.157852 1171 tablet_bootstrap.cc:492] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.147895 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 402) took 2490 ms. Trace:
I20250905 19:52:25.160269 1185 rpcz_store.cc:276] 0905 19:52:22.657071 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.220864 (+563793us) service_pool.cc:225] Handling call
0905 19:52:25.094635 (+1873771us) ts_tablet_manager.cc:1145] Deleting tablet cf2881d228964cd3bef040a06045d8d2
0905 19:52:25.094643 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.119651 (+ 25008us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.130831 (+ 11180us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.147853 (+ 17022us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873532}
I20250905 19:52:25.160740 1185 tablet_replica.cc:331] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.160938 1185 raft_consensus.cc:2241] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.161101 1185 raft_consensus.cc:2270] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.162148 1171 tablet_bootstrap.cc:654] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.163592 1185 ts_tablet_manager.cc:1905] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.164386 1169 tablet_bootstrap.cc:492] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.164662 1170 tablet_bootstrap.cc:654] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.166213 1169 tablet_bootstrap.cc:654] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.169970 1798 raft_consensus.cc:491] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.170186 1798 raft_consensus.cc:513] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.171029 1184 ts_tablet_manager.cc:1918] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.171038 1185 ts_tablet_manager.cc:1918] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.171161 1184 log.cc:1199] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:25.171192 1185 log.cc:1199] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/29c2b030fafd44ad9e648b2527a4ad07
I20250905 19:52:25.171556 1184 ts_tablet_manager.cc:1939] T 2d84653c9e7a4857a7580fa85f8e4732 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.171554 1185 ts_tablet_manager.cc:1939] T 29c2b030fafd44ad9e648b2527a4ad07 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.171730 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:25.172361 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 403) took 2515 ms. Trace:
I20250905 19:52:25.172396 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.172418 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 331) took 4144 ms. Trace:
I20250905 19:52:25.172466 1185 rpcz_store.cc:276] 0905 19:52:22.657165 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.221128 (+563963us) service_pool.cc:225] Handling call
0905 19:52:25.160668 (+1939540us) ts_tablet_manager.cc:1145] Deleting tablet 29c2b030fafd44ad9e648b2527a4ad07
0905 19:52:25.160678 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.166517 (+ 5839us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.170925 (+ 4408us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.172309 (+ 1384us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1939358}
I20250905 19:52:25.172546 1184 rpcz_store.cc:276] 0905 19:52:21.028308 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.028406 (+ 98us) service_pool.cc:225] Handling call
0905 19:52:25.108663 (+4080257us) ts_tablet_manager.cc:1145] Deleting tablet 2d84653c9e7a4857a7580fa85f8e4732
0905 19:52:25.108677 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.115572 (+ 6895us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.170881 (+ 55309us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.172379 (+ 1498us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4072204}
W20250905 19:52:25.172978 984 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.173592 986 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.173717 986 leader_election.cc:304] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.174012 1809 raft_consensus.cc:2747] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.174520 1171 tablet_bootstrap.cc:492] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.174552 1798 leader_election.cc:290] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.174659 1171 ts_tablet_manager.cc:1397] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.003s
I20250905 19:52:25.175427 1171 raft_consensus.cc:357] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.175603 1171 raft_consensus.cc:383] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.175691 1171 raft_consensus.cc:738] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.175933 1171 consensus_queue.cc:260] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.177064 1171 ts_tablet_manager.cc:1428] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.172986 1184 tablet_replica.cc:331] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.177477 1184 raft_consensus.cc:2241] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.177660 1184 raft_consensus.cc:2270] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.178360 1184 ts_tablet_manager.cc:1905] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.178732 1171 tablet_bootstrap.cc:492] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.181047 1171 tablet_bootstrap.cc:654] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.181603 776 raft_consensus.cc:3058] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.181929 1185 tablet_replica.cc:331] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.182183 1185 raft_consensus.cc:2241] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:25.182529 1185 raft_consensus.cc:2270] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.183068 1185 ts_tablet_manager.cc:1905] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.190002 1185 ts_tablet_manager.cc:1918] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:25.190156 1185 log.cc:1199] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/66df0522aa2842f68aa824571fa61b05
I20250905 19:52:25.190567 1185 ts_tablet_manager.cc:1939] T 66df0522aa2842f68aa824571fa61b05 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.191385 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 404) took 2483 ms. Trace:
I20250905 19:52:25.191511 1185 rpcz_store.cc:276] 0905 19:52:22.708338 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.221322 (+512984us) service_pool.cc:225] Handling call
0905 19:52:25.177241 (+1955919us) ts_tablet_manager.cc:1145] Deleting tablet 66df0522aa2842f68aa824571fa61b05
0905 19:52:25.181861 (+ 4620us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.186687 (+ 4826us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.189870 (+ 3183us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.191349 (+ 1479us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1955783}
I20250905 19:52:25.192708 1935 tablet.cc:1722] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:25.196995 1849 raft_consensus.cc:491] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.197171 1849 raft_consensus.cc:513] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.197777 1849 leader_election.cc:290] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:25.198863 712 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
W20250905 19:52:25.200181 711 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:25.200331 711 leader_election.cc:304] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.200748 1849 raft_consensus.cc:2747] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.194478 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2]
I20250905 19:52:25.202088 1170 tablet_bootstrap.cc:492] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.202344 1170 ts_tablet_manager.cc:1397] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.074s user 0.001s sys 0.008s
I20250905 19:52:25.203119 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a]
I20250905 19:52:25.203346 1184 ts_tablet_manager.cc:1918] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:25.203502 1184 log.cc:1199] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/fbab0209a48d4ad8b2f2dec0562e9598
I20250905 19:52:25.203662 1170 raft_consensus.cc:357] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.203876 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]
I20250905 19:52:25.203907 1170 raft_consensus.cc:383] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.203943 1184 ts_tablet_manager.cc:1939] T fbab0209a48d4ad8b2f2dec0562e9598 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.204098 1170 raft_consensus.cc:738] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.204492 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b]
I20250905 19:52:25.204831 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 333) took 4135 ms. Trace:
I20250905 19:52:25.204931 1184 rpcz_store.cc:276] 0905 19:52:21.069462 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.069532 (+ 70us) service_pool.cc:225] Handling call
0905 19:52:25.172908 (+4103376us) ts_tablet_manager.cc:1145] Deleting tablet fbab0209a48d4ad8b2f2dec0562e9598
0905 19:52:25.172921 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.197086 (+ 24165us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.203219 (+ 6133us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.204769 (+ 1550us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4102672}
I20250905 19:52:25.204825 1170 consensus_queue.cc:260] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.205401 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 332) took 4174 ms. Trace:
I20250905 19:52:25.205569 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906]
I20250905 19:52:25.205502 1184 rpcz_store.cc:276] 0905 19:52:21.030892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.091884 (+ 60992us) service_pool.cc:225] Handling call
0905 19:52:25.205232 (+4113348us) ts_tablet_manager.cc:1145] Deleting tablet f48f9f789b6e46ec871a0fd007770dfc
0905 19:52:25.205243 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.205347 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4085973}
I20250905 19:52:25.205874 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 341) took 3872 ms. Trace:
I20250905 19:52:25.205968 1184 rpcz_store.cc:276] 0905 19:52:21.333264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.333338 (+ 74us) service_pool.cc:225] Handling call
0905 19:52:25.205734 (+3872396us) ts_tablet_manager.cc:1145] Deleting tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
0905 19:52:25.205743 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.205825 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3866115}
I20250905 19:52:25.206346 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 343) took 3722 ms. Trace:
I20250905 19:52:25.206423 1184 rpcz_store.cc:276] 0905 19:52:21.483522 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.483577 (+ 55us) service_pool.cc:225] Handling call
0905 19:52:25.206209 (+3722632us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:25.206217 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.206302 (+ 85us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722002}
I20250905 19:52:25.206758 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 342) took 3723 ms. Trace:
I20250905 19:52:25.206833 1184 rpcz_store.cc:276] 0905 19:52:21.482962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.483058 (+ 96us) service_pool.cc:225] Handling call
0905 19:52:25.206614 (+3723556us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:25.206622 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.206711 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722279}
I20250905 19:52:25.206935 1170 ts_tablet_manager.cc:1428] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:25.207695 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 345) took 3710 ms. Trace:
I20250905 19:52:25.207785 1184 rpcz_store.cc:276] 0905 19:52:21.496997 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.504195 (+ 7198us) service_pool.cc:225] Handling call
0905 19:52:25.207067 (+3702872us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:25.207074 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.207146 (+ 72us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3702616}
I20250905 19:52:25.208444 1170 tablet_bootstrap.cc:492] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.209646 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 386) took 2556 ms. Trace:
I20250905 19:52:25.209771 1185 rpcz_store.cc:276] 0905 19:52:22.652865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.793799 (+140934us) service_pool.cc:225] Handling call
0905 19:52:25.209444 (+2415645us) ts_tablet_manager.cc:1145] Deleting tablet 0267ac3e3f1a45978807737eca5d91ef
0905 19:52:25.209455 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.209597 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974525}
I20250905 19:52:25.210646 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6ec4f72266c7447da59b8544cfa98cd6 [id=0b6f3d59aebb4de0b34652b9f2c8ded7]
I20250905 19:52:25.210978 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 400) took 2554 ms. Trace:
I20250905 19:52:25.211107 1185 rpcz_store.cc:276] 0905 19:52:22.656896 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.810941 (+154045us) service_pool.cc:225] Handling call
0905 19:52:25.210809 (+2399868us) ts_tablet_manager.cc:1145] Deleting tablet d296428b70f9411190b6c408d81639f2
0905 19:52:25.210821 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.210938 (+ 117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1975119}
I20250905 19:52:25.212211 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 401) took 2555 ms. Trace:
I20250905 19:52:25.212324 1185 rpcz_store.cc:276] 0905 19:52:22.656981 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.830359 (+173378us) service_pool.cc:225] Handling call
0905 19:52:25.212073 (+2381714us) ts_tablet_manager.cc:1145] Deleting tablet fd1280fefc3e43db9928567a734644f1
0905 19:52:25.212086 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.212176 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1974951}
I20250905 19:52:25.213065 1170 tablet_bootstrap.cc:654] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.213325 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 419) took 1868 ms. Trace:
I20250905 19:52:25.213444 1185 rpcz_store.cc:276] 0905 19:52:23.345316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.441296 (+ 95980us) service_pool.cc:225] Handling call
0905 19:52:25.213205 (+1771909us) ts_tablet_manager.cc:1145] Deleting tablet 3609de83732d4ad8857372acfbb5c32a
0905 19:52:25.213216 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.213295 (+ 79us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771651}
I20250905 19:52:25.213838 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]
I20250905 19:52:25.214388 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 420) took 1865 ms. Trace:
I20250905 19:52:25.214499 1185 rpcz_store.cc:276] 0905 19:52:23.348568 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.441582 (+ 93014us) service_pool.cc:225] Handling call
0905 19:52:25.214265 (+1772683us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:25.214273 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.214353 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1772529}
I20250905 19:52:25.215294 1185 tablet_replica.cc:331] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.215505 1185 raft_consensus.cc:2241] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.215674 1185 raft_consensus.cc:2270] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.216519 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3]
I20250905 19:52:25.216971 1185 ts_tablet_manager.cc:1905] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.217232 1849 raft_consensus.cc:491] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.217418 1849 raft_consensus.cc:513] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.218055 1849 leader_election.cc:290] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:25.219506 712 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
W20250905 19:52:25.219851 711 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:25.220023 711 leader_election.cc:304] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.220552 1849 raft_consensus.cc:2747] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.220769 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba]
I20250905 19:52:25.221592 1935 tablet.cc:1717] T 4fac0e1f483b40ab8d66cb5a9ee808fc P f21c498a0d964e8fbebfd1b3cb1916fe: Skipping requested alter to schema version 2, tablet already version 2
I20250905 19:52:25.221661 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 353) took 3651 ms. Trace:
I20250905 19:52:25.221757 1184 rpcz_store.cc:276] 0905 19:52:21.570553 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.577654 (+ 7101us) service_pool.cc:225] Handling call
0905 19:52:25.221486 (+3643832us) ts_tablet_manager.cc:1145] Deleting tablet 8edf4b54974c427a855eee5792d75292
0905 19:52:25.221496 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.221593 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3642608}
I20250905 19:52:25.221990 1169 tablet_bootstrap.cc:492] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.222147 1169 ts_tablet_manager.cc:1397] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.058s user 0.003s sys 0.004s
I20250905 19:52:25.223042 1184 tablet_replica.cc:331] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.223245 1184 raft_consensus.cc:2241] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.223122 1169 raft_consensus.cc:357] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.223414 1169 raft_consensus.cc:383] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.223491 1169 raft_consensus.cc:738] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.223795 1169 consensus_queue.cc:260] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.223924 1184 raft_consensus.cc:2270] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.224393 1169 ts_tablet_manager.cc:1428] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.224393 1186 ts_tablet_manager.cc:1918] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.224511 1186 log.cc:1199] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/7c5807c1132747e6af79d93e7496d500
I20250905 19:52:25.224647 1184 ts_tablet_manager.cc:1905] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.225525 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c]
I20250905 19:52:25.227131 1186 ts_tablet_manager.cc:1939] T 7c5807c1132747e6af79d93e7496d500 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:25.227850 1169 tablet_bootstrap.cc:492] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.230026 1169 tablet_bootstrap.cc:654] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.230314 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a [id=1598bab0cf8f41e0a4f7ee643cfeeaab]
I20250905 19:52:25.230803 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]
I20250905 19:52:25.232189 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]
I20250905 19:52:25.232708 1171 tablet_bootstrap.cc:492] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.232875 1171 ts_tablet_manager.cc:1397] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.054s user 0.008s sys 0.008s
I20250905 19:52:25.233090 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]
I20250905 19:52:25.233498 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6]
I20250905 19:52:25.233670 1171 raft_consensus.cc:357] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.233860 1171 raft_consensus.cc:383] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.233927 1171 raft_consensus.cc:738] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.234158 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]
I20250905 19:52:25.234237 1171 consensus_queue.cc:260] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.234920 1171 ts_tablet_manager.cc:1428] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:25.235086 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]
I20250905 19:52:25.235253 1171 tablet_bootstrap.cc:492] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.235723 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a]
I20250905 19:52:25.236790 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967]
I20250905 19:52:25.237223 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5]
I20250905 19:52:25.237320 1171 tablet_bootstrap.cc:654] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.237636 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_c30c09bc294d465da486648091481ed1 [id=3118b283e4194c5fa5c5a7e270d14e49]
I20250905 19:52:25.237783 1170 tablet_bootstrap.cc:492] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.237957 1170 ts_tablet_manager.cc:1397] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.004s
I20250905 19:52:25.238047 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]
I20250905 19:52:25.238665 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c]
I20250905 19:52:25.238730 1170 raft_consensus.cc:357] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.238940 1170 raft_consensus.cc:383] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.239022 1170 raft_consensus.cc:738] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.239223 1185 ts_tablet_manager.cc:1918] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:25.239388 1185 log.cc:1199] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/0d4204f61929498188c18529148a8d13
I20250905 19:52:25.239889 1185 ts_tablet_manager.cc:1939] T 0d4204f61929498188c18529148a8d13 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.240123 1170 consensus_queue.cc:260] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.240286 1672 raft_consensus.cc:491] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.240470 1672 raft_consensus.cc:513] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.241174 1170 ts_tablet_manager.cc:1428] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:25.241181 1672 leader_election.cc:290] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.241918 1170 tablet_bootstrap.cc:492] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.241888 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "60a41dbd8782460cb0d3c5f8bd59354f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.243574 1170 tablet_bootstrap.cc:654] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:25.244987 849 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:25.245400 1184 ts_tablet_manager.cc:1918] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.245556 1184 log.cc:1199] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/00d8144bb4f14358af539b7aae326918
I20250905 19:52:25.245890 1184 ts_tablet_manager.cc:1939] T 00d8144bb4f14358af539b7aae326918 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:25.247138 848 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.247293 848 leader_election.cc:304] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.247545 1672 raft_consensus.cc:2747] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.249275 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 357) took 3646 ms. Trace:
I20250905 19:52:25.249440 1184 rpcz_store.cc:276] 0905 19:52:21.602435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.610733 (+ 8298us) service_pool.cc:225] Handling call
0905 19:52:25.222955 (+3612222us) ts_tablet_manager.cc:1145] Deleting tablet 00d8144bb4f14358af539b7aae326918
0905 19:52:25.222963 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.227942 (+ 4979us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.245215 (+ 17273us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.249221 (+ 4006us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":151,"tablet-delete.queue_time_us":3611082}
I20250905 19:52:25.250953 1171 tablet_bootstrap.cc:492] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.251097 1171 ts_tablet_manager.cc:1397] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.004s
I20250905 19:52:25.251763 1171 raft_consensus.cc:357] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.251924 1171 raft_consensus.cc:383] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.252030 1171 raft_consensus.cc:738] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.252308 1171 consensus_queue.cc:260] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.253325 1171 ts_tablet_manager.cc:1428] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:25.253603 1171 tablet_bootstrap.cc:492] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.253765 1170 tablet_bootstrap.cc:492] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.253907 1170 ts_tablet_manager.cc:1397] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250905 19:52:25.254590 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6]
I20250905 19:52:25.254719 1170 raft_consensus.cc:357] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.254884 1170 raft_consensus.cc:383] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.254966 1170 raft_consensus.cc:738] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.255105 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92]
I20250905 19:52:25.255465 1171 tablet_bootstrap.cc:654] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.255988 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]
I20250905 19:52:25.256724 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1]
I20250905 19:52:25.256747 1184 tablet_replica.cc:331] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.256961 1184 raft_consensus.cc:2241] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.257393 1184 raft_consensus.cc:2270] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.257756 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646]
I20250905 19:52:25.257925 1184 ts_tablet_manager.cc:1905] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.260211 1170 consensus_queue.cc:260] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.262126 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 421) took 1898 ms. Trace:
I20250905 19:52:25.262295 1185 rpcz_store.cc:276] 0905 19:52:23.348724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.441751 (+ 93027us) service_pool.cc:225] Handling call
0905 19:52:25.215238 (+1773487us) ts_tablet_manager.cc:1145] Deleting tablet 0d4204f61929498188c18529148a8d13
0905 19:52:25.215246 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.229431 (+ 14185us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.239080 (+ 9649us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.247068 (+ 7988us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1773309}
I20250905 19:52:25.262984 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 422) took 1768 ms. Trace:
I20250905 19:52:25.263131 1185 rpcz_store.cc:276] 0905 19:52:23.494390 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.494835 (+ 445us) service_pool.cc:225] Handling call
0905 19:52:25.262805 (+1767970us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:25.262817 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.262933 (+ 116us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745302}
I20250905 19:52:25.263646 1185 tablet_replica.cc:331] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.263844 1170 ts_tablet_manager.cc:1428] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250905 19:52:25.263986 1185 raft_consensus.cc:2241] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.264242 1185 raft_consensus.cc:2270] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.264360 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]
I20250905 19:52:25.264596 1184 ts_tablet_manager.cc:1918] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.264741 1184 log.cc:1199] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/05ae83eea5f74e9187713e8b82c9a23b
I20250905 19:52:25.264812 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]
I20250905 19:52:25.265082 1185 ts_tablet_manager.cc:1905] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.265113 1184 ts_tablet_manager.cc:1939] T 05ae83eea5f74e9187713e8b82c9a23b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.265853 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 364) took 3645 ms. Trace:
I20250905 19:52:25.265991 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]
I20250905 19:52:25.265956 1184 rpcz_store.cc:276] 0905 19:52:21.620587 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.624837 (+ 4250us) service_pool.cc:225] Handling call
0905 19:52:25.256647 (+3631810us) ts_tablet_manager.cc:1145] Deleting tablet 05ae83eea5f74e9187713e8b82c9a23b
0905 19:52:25.256662 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.261475 (+ 4813us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.264489 (+ 3014us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.265798 (+ 1309us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3623638}
I20250905 19:52:25.266448 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]
I20250905 19:52:25.267637 1170 tablet_bootstrap.cc:492] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.269377 1169 tablet_bootstrap.cc:492] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.269460 1184 tablet_replica.cc:331] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.269572 1169 ts_tablet_manager.cc:1397] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.042s user 0.007s sys 0.005s
I20250905 19:52:25.270367 1170 tablet_bootstrap.cc:654] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.270392 1169 raft_consensus.cc:357] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.270555 1169 raft_consensus.cc:383] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.270604 1171 tablet_bootstrap.cc:492] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.270632 1169 raft_consensus.cc:738] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.270788 1171 ts_tablet_manager.cc:1397] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.003s
I20250905 19:52:25.270900 1169 consensus_queue.cc:260] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.271559 1171 raft_consensus.cc:357] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.271754 1171 raft_consensus.cc:383] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.271823 1171 raft_consensus.cc:738] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.272130 1171 consensus_queue.cc:260] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.272701 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]
I20250905 19:52:25.276844 1171 ts_tablet_manager.cc:1428] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:25.277168 1171 tablet_bootstrap.cc:492] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.277904 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad]
I20250905 19:52:25.277935 1672 raft_consensus.cc:491] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.278110 1672 raft_consensus.cc:513] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.278439 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_2cde62a9512e4445b5d0eb12119380f1 [id=46715848277847218d557157e4dfc77b]
I20250905 19:52:25.278837 1672 leader_election.cc:290] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.278889 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e]
I20250905 19:52:25.278899 1171 tablet_bootstrap.cc:654] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.279448 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]
I20250905 19:52:25.279569 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.279954 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:25.280200 848 leader_election.cc:343] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.280211 1050 raft_consensus.cc:2466] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:25.280483 1184 raft_consensus.cc:2241] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.280510 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]
I20250905 19:52:25.280675 1184 raft_consensus.cc:2270] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.281087 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]
I20250905 19:52:25.281226 1184 ts_tablet_manager.cc:1905] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.281240 849 leader_election.cc:304] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.285492 1672 raft_consensus.cc:2802] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.285601 1672 raft_consensus.cc:491] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.285712 1672 raft_consensus.cc:3058] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.288499 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f]
I20250905 19:52:25.288518 1672 raft_consensus.cc:513] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.289044 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]
I20250905 19:52:25.289495 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]
I20250905 19:52:25.289877 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52]
I20250905 19:52:25.291015 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a]
I20250905 19:52:25.291615 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]
I20250905 19:52:25.292133 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]
I20250905 19:52:25.292599 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:25.292783 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_cb111dc4aa3946e8a89529d1cd623c08 [id=5ac15645cb4d4730a4b4464a85299bed]
I20250905 19:52:25.293004 1798 raft_consensus.cc:491] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:25.293125 848 leader_election.cc:343] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.293180 1798 raft_consensus.cc:513] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.293856 1798 leader_election.cc:290] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.293896 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]
I20250905 19:52:25.294366 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:25.294627 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:25.294752 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.294796 1050 raft_consensus.cc:3058] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:25.295341 986 leader_election.cc:343] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.295704 984 leader_election.cc:343] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.295814 984 leader_election.cc:304] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.296295 1798 raft_consensus.cc:2747] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.297590 1050 raft_consensus.cc:2466] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.298075 849 leader_election.cc:304] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.298400 1584 raft_consensus.cc:2802] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.298557 1584 raft_consensus.cc:695] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.298661 1171 tablet_bootstrap.cc:492] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.298843 1171 ts_tablet_manager.cc:1397] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250905 19:52:25.298870 1584 consensus_queue.cc:237] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.298985 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]
I20250905 19:52:25.299458 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1]
I20250905 19:52:25.299820 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]
I20250905 19:52:25.301184 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]
I20250905 19:52:25.301120 1171 raft_consensus.cc:357] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.301366 1171 raft_consensus.cc:383] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.301486 1672 leader_election.cc:290] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.301496 1171 raft_consensus.cc:738] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.302150 1171 consensus_queue.cc:260] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.303385 1171 ts_tablet_manager.cc:1428] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:25.303560 1170 tablet_bootstrap.cc:492] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.303660 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]
I20250905 19:52:25.303704 1171 tablet_bootstrap.cc:492] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.303738 1170 ts_tablet_manager.cc:1397] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.009s
I20250905 19:52:25.304500 1170 raft_consensus.cc:357] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.304695 1170 raft_consensus.cc:383] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.304777 1170 raft_consensus.cc:738] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.304924 1169 ts_tablet_manager.cc:1428] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.035s user 0.001s sys 0.001s
I20250905 19:52:25.305030 1170 consensus_queue.cc:260] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.306012 1171 tablet_bootstrap.cc:654] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.306910 1170 ts_tablet_manager.cc:1428] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:25.307214 1170 tablet_bootstrap.cc:492] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.307571 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]
I20250905 19:52:25.308923 1170 tablet_bootstrap.cc:654] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.309496 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499]
I20250905 19:52:25.310490 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9]
I20250905 19:52:25.312243 1184 ts_tablet_manager.cc:1918] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.312367 1184 log.cc:1199] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:25.312671 1184 ts_tablet_manager.cc:1939] T f28e403dc2f6471ebea1805fd843cfb1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.313046 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b]
I20250905 19:52:25.313392 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 363) took 3692 ms. Trace:
I20250905 19:52:25.313587 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef]
I20250905 19:52:25.313520 1184 rpcz_store.cc:276] 0905 19:52:21.620431 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:21.627244 (+ 6813us) service_pool.cc:225] Handling call
0905 19:52:25.269354 (+3642110us) ts_tablet_manager.cc:1145] Deleting tablet f28e403dc2f6471ebea1805fd843cfb1
0905 19:52:25.269367 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.289542 (+ 20175us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.312103 (+ 22561us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.313356 (+ 1253us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3635831}
I20250905 19:52:25.314656 1184 tablet_replica.cc:331] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.314880 1184 raft_consensus.cc:2241] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.315095 1184 raft_consensus.cc:2270] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.315666 1184 ts_tablet_manager.cc:1905] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.318773 776 raft_consensus.cc:3058] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.319458 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]
I20250905 19:52:25.319955 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9]
I20250905 19:52:25.320454 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]
I20250905 19:52:25.320932 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6]
I20250905 19:52:25.321389 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]
I20250905 19:52:25.322152 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]
I20250905 19:52:25.323025 1170 tablet_bootstrap.cc:492] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.323184 1170 ts_tablet_manager.cc:1397] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.000s
I20250905 19:52:25.324100 1184 ts_tablet_manager.cc:1918] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:25.324245 1184 log.cc:1199] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/2f581403ec924fb0948400df41ed8aab
I20250905 19:52:25.324069 1170 raft_consensus.cc:357] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.324333 1170 raft_consensus.cc:383] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.324436 1170 raft_consensus.cc:738] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.324661 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]
I20250905 19:52:25.324692 1184 ts_tablet_manager.cc:1939] T 2f581403ec924fb0948400df41ed8aab P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.324697 1170 consensus_queue.cc:260] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.325340 1170 ts_tablet_manager.cc:1428] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:25.325474 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 367) took 3221 ms. Trace:
I20250905 19:52:25.325634 1170 tablet_bootstrap.cc:492] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.325572 1184 rpcz_store.cc:276] 0905 19:52:22.104024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.111569 (+ 7545us) service_pool.cc:225] Handling call
0905 19:52:25.314252 (+3202683us) ts_tablet_manager.cc:1145] Deleting tablet 2f581403ec924fb0948400df41ed8aab
0905 19:52:25.314577 (+ 325us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.320876 (+ 6299us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.323813 (+ 2937us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.325420 (+ 1607us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3202344}
I20250905 19:52:25.326197 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]
I20250905 19:52:25.326227 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 366) took 3222 ms. Trace:
I20250905 19:52:25.326321 1184 rpcz_store.cc:276] 0905 19:52:22.103843 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.112063 (+ 8220us) service_pool.cc:225] Handling call
0905 19:52:25.326056 (+3213993us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:25.326068 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.326177 (+ 109us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3213693}
I20250905 19:52:25.326790 1184 tablet_replica.cc:331] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.327003 1184 raft_consensus.cc:2241] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.327190 1184 raft_consensus.cc:2270] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.327662 1170 tablet_bootstrap.cc:654] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.327858 1184 ts_tablet_manager.cc:1905] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.328290 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482]
I20250905 19:52:25.329099 1169 tablet_bootstrap.cc:492] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.329634 1171 tablet_bootstrap.cc:492] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.329810 1171 ts_tablet_manager.cc:1397] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.002s
I20250905 19:52:25.330544 1171 raft_consensus.cc:357] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.330751 1171 raft_consensus.cc:383] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.330823 1171 raft_consensus.cc:738] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.331095 1171 consensus_queue.cc:260] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.331609 1171 ts_tablet_manager.cc:1428] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:25.331921 1171 tablet_bootstrap.cc:492] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.333626 1171 tablet_bootstrap.cc:654] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.330691 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde]
I20250905 19:52:25.334333 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]
I20250905 19:52:25.335074 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_849b4e364b3f4ff2b47a2347f84d6b0d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]
I20250905 19:52:25.340431 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc]
I20250905 19:52:25.341208 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c]
I20250905 19:52:25.342033 1849 raft_consensus.cc:491] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.342216 1849 raft_consensus.cc:513] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.342934 1849 leader_election.cc:290] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.343307 1170 tablet_bootstrap.cc:492] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.343531 1170 ts_tablet_manager.cc:1397] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
W20250905 19:52:25.344223 712 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:25.344440 1170 raft_consensus.cc:357] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.344512 711 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:25.344748 711 leader_election.cc:304] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.345191 1849 raft_consensus.cc:2747] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.345423 1170 raft_consensus.cc:383] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.345543 1170 raft_consensus.cc:738] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.345829 1170 consensus_queue.cc:260] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.346580 1170 ts_tablet_manager.cc:1428] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:25.346900 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]
I20250905 19:52:25.349293 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]
I20250905 19:52:25.350234 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]
I20250905 19:52:25.350926 1170 tablet_bootstrap.cc:492] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.352471 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6]
I20250905 19:52:25.352779 1170 tablet_bootstrap.cc:654] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.342296 1184 ts_tablet_manager.cc:1918] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.353837 1184 log.cc:1199] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:25.354188 1184 ts_tablet_manager.cc:1939] T 0267ac3e3f1a45978807737eca5d91ef P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.354959 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]
I20250905 19:52:25.355032 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 387) took 3117 ms. Trace:
I20250905 19:52:25.355162 1184 rpcz_store.cc:276] 0905 19:52:22.237426 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.237492 (+ 66us) service_pool.cc:225] Handling call
0905 19:52:25.326709 (+3089217us) ts_tablet_manager.cc:1145] Deleting tablet 0267ac3e3f1a45978807737eca5d91ef
0905 19:52:25.326719 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.331180 (+ 4461us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.342115 (+ 10935us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.354998 (+ 12883us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3088957}
I20250905 19:52:25.355716 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 386) took 3119 ms. Trace:
I20250905 19:52:25.355826 1184 rpcz_store.cc:276] 0905 19:52:22.236559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.236631 (+ 72us) service_pool.cc:225] Handling call
0905 19:52:25.355571 (+3118940us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:25.355582 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.355689 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3116611}
I20250905 19:52:25.356227 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9da2eedb50e64356ba6a49d93bafbe28 [id=92aa73bcd0ad43d69787ac8ed44d6a07]
I20250905 19:52:25.356288 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 388) took 3097 ms. Trace:
I20250905 19:52:25.356446 1184 rpcz_store.cc:276] 0905 19:52:22.258937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.259103 (+ 166us) service_pool.cc:225] Handling call
0905 19:52:25.356169 (+3097066us) ts_tablet_manager.cc:1145] Deleting tablet 24bcd21f19ba465aa550102e0fccb674
0905 19:52:25.356178 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.356260 (+ 82us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3084591}
I20250905 19:52:25.356806 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09]
I20250905 19:52:25.356840 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 365) took 3264 ms. Trace:
I20250905 19:52:25.356943 1184 rpcz_store.cc:276] 0905 19:52:22.092755 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.239470 (+146715us) service_pool.cc:225] Handling call
0905 19:52:25.356742 (+3117272us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:25.356750 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.356816 (+ 66us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3084556}
I20250905 19:52:25.357278 1184 tablet_replica.cc:331] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.357497 1184 raft_consensus.cc:2241] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.357623 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_a3f620901eff44dea30bf9e4de3c5552 [id=93e1f25857f644c5bd22e445576390a7]
I20250905 19:52:25.357656 1184 raft_consensus.cc:2270] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.358179 1184 ts_tablet_manager.cc:1905] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.358547 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b]
I20250905 19:52:25.359359 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18]
I20250905 19:52:25.359789 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]
I20250905 19:52:25.360267 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9]
I20250905 19:52:25.360738 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4447fb37960e42dfa2c6eb8ced9c6b68 [id=99384aac4e624d6db630b249c3ba2543]
I20250905 19:52:25.361394 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]
I20250905 19:52:25.361812 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]
I20250905 19:52:25.361884 1171 tablet_bootstrap.cc:492] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.362058 1171 ts_tablet_manager.cc:1397] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.030s user 0.007s sys 0.011s
I20250905 19:52:25.362689 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778]
I20250905 19:52:25.363087 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]
I20250905 19:52:25.362982 1171 raft_consensus.cc:357] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.363175 1171 raft_consensus.cc:383] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.363277 1171 raft_consensus.cc:738] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.363536 1171 consensus_queue.cc:260] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.363863 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959]
I20250905 19:52:25.330951 1169 tablet_bootstrap.cc:654] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.364078 1171 ts_tablet_manager.cc:1428] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250905 19:52:25.364375 1171 tablet_bootstrap.cc:492] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.364507 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]
I20250905 19:52:25.365259 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]
I20250905 19:52:25.365708 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]
I20250905 19:52:25.366075 1171 tablet_bootstrap.cc:654] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.366173 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30]
I20250905 19:52:25.366585 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3]
I20250905 19:52:25.367017 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_191f4f9b06184a8a8429cce5802c520e [id=a9604dfc1b44424f97da39857b13ae91]
I20250905 19:52:25.368067 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d]
I20250905 19:52:25.368779 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d]
I20250905 19:52:25.369277 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]
I20250905 19:52:25.370043 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]
I20250905 19:52:25.370970 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d]
I20250905 19:52:25.371665 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d]
I20250905 19:52:25.372359 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6]
I20250905 19:52:25.373232 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]
I20250905 19:52:25.373688 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_19682dbce0bc4480b82d39235ba7f55b [id=bff29fe995d34f32ad9780ad5591a8b2]
I20250905 19:52:25.374132 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5]
I20250905 19:52:25.374594 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_ad054f6da70449ecad529f69c6aa9705 [id=c31b7e937b6d4adfb299ae8e46330d55]
I20250905 19:52:25.375348 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]
I20250905 19:52:25.376623 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]
I20250905 19:52:25.377288 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f]
I20250905 19:52:25.377790 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]
I20250905 19:52:25.378311 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2]
I20250905 19:52:25.378757 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]
I20250905 19:52:25.379375 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb]
I20250905 19:52:25.380259 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]
I20250905 19:52:25.380726 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]
I20250905 19:52:25.381422 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03]
I20250905 19:52:25.382119 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]
I20250905 19:52:25.383178 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1]
I20250905 19:52:25.384168 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_5bf03647f71d46a8b3d3f0b5bc385154 [id=e19dba4ecf6d4c22965720b89e7c45d4]
I20250905 19:52:25.384635 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]
W20250905 19:52:25.386857 439 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f953a7490434bc7afa5510c33e79401 consensus state has no leader
I20250905 19:52:25.389050 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]
W20250905 19:52:25.389199 485 rpcz_store.cc:267] Call kudu.server.GenericService.CheckLeaks from 127.0.0.1:52570 (request call id 31) took 880 ms (client timeout 1000 ms). Trace:
W20250905 19:52:25.389338 485 rpcz_store.cc:269] 0905 19:52:24.508401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.508532 (+ 131us) service_pool.cc:225] Handling call
0905 19:52:25.389157 (+880625us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:25.389695 450 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9067a9bea2cc48ecab23b9fd1749474c" } modify_external_catalogs: true
I20250905 19:52:25.389744 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 3 candidate_status { last_received { term: 2 index: 593 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:25.389901 450 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_9067a9bea2cc48ecab23b9fd1749474c" } modify_external_catalogs: true
I20250905 19:52:25.390599 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1]
I20250905 19:52:25.391106 464 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 3 candidate_status { last_received { term: 2 index: 593 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:25.391185 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 0edac849e8834894be2d4ff323edd13d (table default.Table_a39be9af778c4fa49c2376cb4bd7397b [id=e9ed839c21da4467a0f165e7fedbe966]) successfully deleted
I20250905 19:52:25.391382 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 3602094b96c14e66b997e8035fc53c40 (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]) successfully deleted
I20250905 19:52:25.391564 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d480531ba94f4b0f9fe40e88fe91512e (table default.Table_c7b1f5de2f2f4657971e14388d57ef5a [id=d92e1dc4017f4f8cbfdeeb40900f136c]) successfully deleted
I20250905 19:52:25.391709 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 36f966bf512d4cbe9ae406386db31533 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]) successfully deleted
I20250905 19:52:25.391930 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 2c884954e5dc4445a12252953593d0bc (table default.Table_b42374e796034b1cbfc8659587477d3c [id=26cde3c02a704bb4931f65f8dba00173]) successfully deleted
I20250905 19:52:25.392062 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]
W20250905 19:52:25.392154 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet b13d282e1dc54d1faa2882c9b39d17cb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet b13d282e1dc54d1faa2882c9b39d17cb already in progress: creating tablet
W20250905 19:52:25.392362 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet f48f9f789b6e46ec871a0fd007770dfc because tablet deleting was already in progress. No further retry: Already present: State transition of tablet f48f9f789b6e46ec871a0fd007770dfc already in progress: creating tablet
I20250905 19:52:25.392505 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 2d84653c9e7a4857a7580fa85f8e4732 (table default.Table_48c2edbf02444d7580fff76f96c9c655 [id=12633ed634a44d74830d55d4e7191808]) successfully deleted
I20250905 19:52:25.392700 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet fbab0209a48d4ad8b2f2dec0562e9598 (table default.Table_89a831c4cad3457b86b01609bfa3a57b [id=19d87a3b106f49deb7e206bfdcbaa041]) successfully deleted
W20250905 19:52:25.392933 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e already in progress: creating tablet
W20250905 19:52:25.393179 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 06850faa25de4b3d83f6bf98ccd370f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06850faa25de4b3d83f6bf98ccd370f0 already in progress: creating tablet
W20250905 19:52:25.393381 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 54628965b82e470d811eff95ffc56c5f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 54628965b82e470d811eff95ffc56c5f already in progress: creating tablet
W20250905 19:52:25.393628 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8edf4b54974c427a855eee5792d75292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8edf4b54974c427a855eee5792d75292 already in progress: creating tablet
W20250905 19:52:25.393828 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c99a76802b481c98f77dbd90de7cbb already in progress: creating tablet
I20250905 19:52:25.394028 442 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 2f581403ec924fb0948400df41ed8aab (table default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]) successfully deleted
W20250905 19:52:25.394240 442 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d38452d3a0641ffa4af5d935f8ad9f0 already in progress: creating tablet
I20250905 19:52:25.394734 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]
I20250905 19:52:25.395258 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244]
I20250905 19:52:25.395730 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816]
I20250905 19:52:25.396188 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]
I20250905 19:52:25.396572 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]
I20250905 19:52:25.397135 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39]
I20250905 19:52:25.397487 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4]
I20250905 19:52:25.397898 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801]
I20250905 19:52:25.398247 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]
I20250905 19:52:25.399120 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]
I20250905 19:52:25.399937 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]
I20250905 19:52:25.400498 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]
W20250905 19:52:25.401162 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3609de83732d4ad8857372acfbb5c32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3609de83732d4ad8857372acfbb5c32a already in progress: creating tablet
W20250905 19:52:25.401381 440 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 814221us
I20250905 19:52:25.402128 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]
W20250905 19:52:25.402397 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6ebab2e3698f4f60a770bde9cc019600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ebab2e3698f4f60a770bde9cc019600 already in progress: creating tablet
I20250905 19:52:25.402508 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147]
W20250905 19:52:25.402664 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0d4204f61929498188c18529148a8d13 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 0d4204f61929498188c18529148a8d13 already in progress: creating tablet
I20250905 19:52:25.402843 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac]
W20250905 19:52:25.402892 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet c46ea498d6a14d1ebdffea415cfc7114 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet c46ea498d6a14d1ebdffea415cfc7114 already in progress: creating tablet
W20250905 19:52:25.403133 440 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a2ff206366ad4344b4df5fc6a5584f8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ff206366ad4344b4df5fc6a5584f8a already in progress: creating tablet
I20250905 19:52:25.403213 1934 catalog_manager.cc:671] Loaded metadata for table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]
I20250905 19:52:25.405652 465 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e38333af29db4bf6af41ee5a25d6701e" candidate_term: 3 candidate_status { last_received { term: 2 index: 593 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711"
I20250905 19:52:25.414482 455 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:52586 (request call id 550) took 1142 ms. Trace:
I20250905 19:52:25.414788 449 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d66d859b633f453393a574fa8885329b" } modify_external_catalogs: true
I20250905 19:52:25.414994 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu with pid 424
I20250905 19:52:25.414975 449 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:52586:
table { table_name: "default.Table_d66d859b633f453393a574fa8885329b" } modify_external_catalogs: true
I20250905 19:52:25.416208 1934 tablet_loader.cc:96] loaded metadata for tablet 0039b8f0ba704907b2820c9e3a69cd73 (table default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a])
I20250905 19:52:25.418228 1934 tablet_loader.cc:96] loaded metadata for tablet 03a12cf719f34096bb82f5846bbe3b2d (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46])
I20250905 19:52:25.418751 1934 tablet_loader.cc:96] loaded metadata for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a])
I20250905 19:52:25.419826 1934 tablet_loader.cc:96] loaded metadata for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e])
I20250905 19:52:25.420619 1934 tablet_loader.cc:96] loaded metadata for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a [id=d7c0f44965694f77a8eae2ec4dd18a03])
I20250905 19:52:25.424757 1934 tablet_loader.cc:96] loaded metadata for tablet 0d93a328d4934e5294ec66f739eb240b (table default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2])
I20250905 19:52:25.425338 1170 tablet_bootstrap.cc:492] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.425560 1170 ts_tablet_manager.cc:1397] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.000s
I20250905 19:52:25.426585 1934 tablet_loader.cc:96] loaded metadata for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3 (table default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147])
I20250905 19:52:25.426527 1170 raft_consensus.cc:357] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.426785 1170 raft_consensus.cc:383] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.426913 1170 raft_consensus.cc:738] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.427207 1170 consensus_queue.cc:260] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.432782 1934 tablet_loader.cc:96] loaded metadata for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e])
I20250905 19:52:25.433647 1184 ts_tablet_manager.cc:1918] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.434639 1169 tablet_bootstrap.cc:492] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.435546 1934 tablet_loader.cc:96] loaded metadata for tablet 160a7814abf9423dbe965c252c20fa7f (table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00])
I20250905 19:52:25.435835 1169 ts_tablet_manager.cc:1397] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.107s user 0.002s sys 0.002s
I20250905 19:52:25.436828 1169 raft_consensus.cc:357] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.437943 1934 tablet_loader.cc:96] loaded metadata for tablet 180437d1534946619384307ddcb7c937 (table default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc])
I20250905 19:52:25.438128 1169 raft_consensus.cc:383] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.438411 1184 log.cc:1199] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/29c2b030fafd44ad9e648b2527a4ad07
I20250905 19:52:25.438941 1184 ts_tablet_manager.cc:1939] T 29c2b030fafd44ad9e648b2527a4ad07 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.438961 1934 tablet_loader.cc:96] loaded metadata for tablet 1bd731a303a545f492b19f774520582c (table default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244])
I20250905 19:52:25.439841 1934 tablet_loader.cc:96] loaded metadata for tablet 1d095804b51b4205a23ac3d3106312f2 (table default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde])
I20250905 19:52:25.440007 1169 raft_consensus.cc:738] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.440412 1169 consensus_queue.cc:260] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.440857 1934 tablet_loader.cc:96] loaded metadata for tablet 1e9b7458f96148b08962a9b82223187e (table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311])
I20250905 19:52:25.441908 1934 tablet_loader.cc:96] loaded metadata for tablet 2350bd08f977424ba94ee8fa01f66efa (table default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5])
I20250905 19:52:25.442158 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 402) took 2936 ms. Trace:
I20250905 19:52:25.442274 1184 rpcz_store.cc:276] 0905 19:52:22.505802 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.505858 (+ 56us) service_pool.cc:225] Handling call
0905 19:52:25.357228 (+2851370us) ts_tablet_manager.cc:1145] Deleting tablet 29c2b030fafd44ad9e648b2527a4ad07
0905 19:52:25.357236 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.361316 (+ 4080us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.433509 (+ 72193us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.442096 (+ 8587us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2850190}
I20250905 19:52:25.443022 1934 tablet_loader.cc:96] loaded metadata for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73])
I20250905 19:52:25.443189 1184 tablet_replica.cc:331] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.443511 1184 raft_consensus.cc:2241] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.443652 1934 tablet_loader.cc:96] loaded metadata for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_c30c09bc294d465da486648091481ed1 [id=3118b283e4194c5fa5c5a7e270d14e49])
I20250905 19:52:25.443838 1184 raft_consensus.cc:2270] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.444576 1934 tablet_loader.cc:96] loaded metadata for tablet 2568b9f6f37b48efbe92cef4c19d5dfb (table default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b])
I20250905 19:52:25.444643 1184 ts_tablet_manager.cc:1905] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.448441 1934 tablet_loader.cc:96] loaded metadata for tablet 31e332cdfedd4ff6b5026baa73e2caf3 (table default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30])
I20250905 19:52:25.449054 1185 ts_tablet_manager.cc:1918] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.449239 1934 tablet_loader.cc:96] loaded metadata for tablet 32fa16972f2a42d39b338b833c185b28 (table default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959])
I20250905 19:52:25.449280 1185 log.cc:1199] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:25.449748 1185 ts_tablet_manager.cc:1939] T 60a41dbd8782460cb0d3c5f8bd59354f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.449803 1934 tablet_loader.cc:96] loaded metadata for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40])
I20250905 19:52:25.450310 1934 tablet_loader.cc:96] loaded metadata for tablet 3587780819704008b7f4ee6abc714db6 (table default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1])
I20250905 19:52:25.450680 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 425) took 1923 ms. Trace:
I20250905 19:52:25.450866 1185 rpcz_store.cc:276] 0905 19:52:23.527018 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.527221 (+ 203us) service_pool.cc:225] Handling call
0905 19:52:25.263566 (+1736345us) ts_tablet_manager.cc:1145] Deleting tablet 60a41dbd8782460cb0d3c5f8bd59354f
0905 19:52:25.263574 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.377778 (+114204us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.448873 (+ 71095us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.450627 (+ 1754us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1736160}
I20250905 19:52:25.451655 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 423) took 1935 ms. Trace:
I20250905 19:52:25.452236 1170 ts_tablet_manager.cc:1428] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250905 19:52:25.454259 1170 tablet_bootstrap.cc:492] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.456179 1170 tablet_bootstrap.cc:654] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.457998 1169 ts_tablet_manager.cc:1428] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.022s user 0.000s sys 0.002s
I20250905 19:52:25.458401 1169 tablet_bootstrap.cc:492] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.458597 1934 tablet_loader.cc:96] loaded metadata for tablet 37311372c03644b0a99ce92b5d42bfd1 (table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c])
I20250905 19:52:25.459000 1849 raft_consensus.cc:491] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.459223 1934 tablet_loader.cc:96] loaded metadata for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (table default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c])
I20250905 19:52:25.459177 1849 raft_consensus.cc:513] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.459858 1849 leader_election.cc:290] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.460088 1934 tablet_loader.cc:96] loaded metadata for tablet 3b53259c71d5443ebb12827ee78038fe (table default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3])
I20250905 19:52:25.460551 1934 tablet_loader.cc:96] loaded metadata for tablet 3c824f5591044c4198a51a25661f71e5 (table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62])
I20250905 19:52:25.460527 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d4204f61929498188c18529148a8d13" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.461005 1934 tablet_loader.cc:96] loaded metadata for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419 (table default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1])
I20250905 19:52:25.461617 1934 tablet_loader.cc:96] loaded metadata for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 (table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e])
W20250905 19:52:25.461690 712 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 0d4204f61929498188c18529148a8d13
I20250905 19:52:25.461862 712 leader_election.cc:304] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.462138 1934 tablet_loader.cc:96] loaded metadata for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348])
I20250905 19:52:25.462486 1934 tablet_loader.cc:96] loaded metadata for tablet 414f30a772524c33a74864ff741c865f (table default.Table_a3f620901eff44dea30bf9e4de3c5552 [id=93e1f25857f644c5bd22e445576390a7])
I20250905 19:52:25.462909 1934 tablet_loader.cc:96] loaded metadata for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_9da2eedb50e64356ba6a49d93bafbe28 [id=92aa73bcd0ad43d69787ac8ed44d6a07])
I20250905 19:52:25.463485 1934 tablet_loader.cc:96] loaded metadata for tablet 421b28d047124222ae9cbc603ec22cee (table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a])
I20250905 19:52:25.464439 1849 raft_consensus.cc:2747] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.465140 1934 tablet_loader.cc:96] loaded metadata for tablet 48be704b238e481089c727b3e1b96a2f (table default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6])
I20250905 19:52:25.465600 1169 tablet_bootstrap.cc:654] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.465592 1934 tablet_loader.cc:96] loaded metadata for tablet 4a518797d26140f5b9abe41eab415040 (table default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3])
I20250905 19:52:25.466663 1185 rpcz_store.cc:276] 0905 19:52:23.516570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.516785 (+ 215us) service_pool.cc:225] Handling call
0905 19:52:25.451450 (+1934665us) ts_tablet_manager.cc:1145] Deleting tablet a2ff206366ad4344b4df5fc6a5584f8a
0905 19:52:25.451460 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.451586 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923802}
I20250905 19:52:25.467087 1185 tablet_replica.cc:331] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.467322 1934 tablet_loader.cc:96] loaded metadata for tablet 4d6f2e44855d4312b1c0367145e3cee8 (table default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9])
I20250905 19:52:25.467377 1185 raft_consensus.cc:2241] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.467566 1185 raft_consensus.cc:2270] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.467872 1934 tablet_loader.cc:96] loaded metadata for tablet 4dd2a96b8180438fbc722be1c18a0189 (table default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6])
I20250905 19:52:25.468233 1185 ts_tablet_manager.cc:1905] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.468429 1934 tablet_loader.cc:96] loaded metadata for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9])
I20250905 19:52:25.460111 1171 tablet_bootstrap.cc:492] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.469059 1171 ts_tablet_manager.cc:1397] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.105s user 0.029s sys 0.031s
I20250905 19:52:25.469314 1934 tablet_loader.cc:96] loaded metadata for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31])
I20250905 19:52:25.470122 1943 tablet.cc:1722] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:25.470230 1171 raft_consensus.cc:357] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.470455 1171 raft_consensus.cc:383] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.470528 1171 raft_consensus.cc:738] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.470778 1171 consensus_queue.cc:260] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.471354 1171 ts_tablet_manager.cc:1428] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:25.472013 1171 tablet_bootstrap.cc:492] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.473906 1171 tablet_bootstrap.cc:654] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.476069 1170 tablet_bootstrap.cc:492] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.476253 1170 ts_tablet_manager.cc:1397] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.002s
I20250905 19:52:25.477058 1170 raft_consensus.cc:357] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.477284 1170 raft_consensus.cc:383] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.477370 1170 raft_consensus.cc:738] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.479115 1934 tablet_loader.cc:96] loaded metadata for tablet 5302b79208494e48a5f2778c5f52ad01 (table default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1])
I20250905 19:52:25.479887 1934 tablet_loader.cc:96] loaded metadata for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d])
I20250905 19:52:25.480744 1934 tablet_loader.cc:96] loaded metadata for tablet 55cecf59d3f24db88bb10b8e12c12f82 (table default.Table_2cde62a9512e4445b5d0eb12119380f1 [id=46715848277847218d557157e4dfc77b])
I20250905 19:52:25.481211 1934 tablet_loader.cc:96] loaded metadata for tablet 56e7ab709fe04479a7b1f17dba2fd603 (table default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34])
I20250905 19:52:25.484290 1934 tablet_loader.cc:96] loaded metadata for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_cb111dc4aa3946e8a89529d1cd623c08 [id=5ac15645cb4d4730a4b4464a85299bed])
I20250905 19:52:25.484622 1170 consensus_queue.cc:260] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.487275 1934 tablet_loader.cc:96] loaded metadata for tablet 5a59d7fb5dc54b479d965282b1c7d24b (table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d])
I20250905 19:52:25.487541 1170 ts_tablet_manager.cc:1428] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:25.487941 1170 tablet_bootstrap.cc:492] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.488487 1934 tablet_loader.cc:96] loaded metadata for tablet 5aff2acc3f8245e1bca41f971f8e74ce (table default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a])
I20250905 19:52:25.489207 1171 tablet_bootstrap.cc:492] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.489379 1171 ts_tablet_manager.cc:1397] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20250905 19:52:25.490182 1171 raft_consensus.cc:357] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.490376 1171 raft_consensus.cc:383] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.490460 1171 raft_consensus.cc:738] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.490821 1169 tablet_bootstrap.cc:492] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.490736 1171 consensus_queue.cc:260] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.490973 1169 ts_tablet_manager.cc:1397] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.001s
I20250905 19:52:25.491766 1169 raft_consensus.cc:357] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.491930 1169 raft_consensus.cc:383] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.492045 1169 raft_consensus.cc:738] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.492337 1169 consensus_queue.cc:260] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.496759 1171 ts_tablet_manager.cc:1428] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:25.497516 1169 ts_tablet_manager.cc:1428] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:25.497720 1934 tablet_loader.cc:96] loaded metadata for tablet 5b4f9e12781a46f19b563746a70b59e4 (table default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778])
I20250905 19:52:25.498471 1934 tablet_loader.cc:96] loaded metadata for tablet 5cbd471695e748c9831f9b0f6795aea7 (table default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a])
I20250905 19:52:25.499245 1170 tablet_bootstrap.cc:654] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.499423 1169 tablet_bootstrap.cc:492] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.500957 1171 tablet_bootstrap.cc:492] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.501246 1169 tablet_bootstrap.cc:654] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.502550 1934 tablet_loader.cc:96] loaded metadata for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50])
I20250905 19:52:25.503019 1934 tablet_loader.cc:96] loaded metadata for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d])
I20250905 19:52:25.503674 1934 tablet_loader.cc:96] loaded metadata for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f])
I20250905 19:52:25.500959 1184 ts_tablet_manager.cc:1918] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:25.503933 1185 ts_tablet_manager.cc:1918] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.503954 1184 log.cc:1199] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/cf2881d228964cd3bef040a06045d8d2
I20250905 19:52:25.504446 1184 ts_tablet_manager.cc:1939] T cf2881d228964cd3bef040a06045d8d2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.505178 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 401) took 3002 ms. Trace:
I20250905 19:52:25.505301 1184 rpcz_store.cc:276] 0905 19:52:22.502598 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.502914 (+ 316us) service_pool.cc:225] Handling call
0905 19:52:25.443023 (+2940109us) ts_tablet_manager.cc:1145] Deleting tablet cf2881d228964cd3bef040a06045d8d2
0905 19:52:25.443036 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.479110 (+ 36074us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.500799 (+ 21689us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.505147 (+ 4348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2933217}
I20250905 19:52:25.505983 1934 tablet_loader.cc:96] loaded metadata for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1])
I20250905 19:52:25.506227 1185 log.cc:1199] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:25.506708 1934 tablet_loader.cc:96] loaded metadata for tablet 65368306f86845008d4d486023e1fc29 (table default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6])
I20250905 19:52:25.506719 1185 ts_tablet_manager.cc:1939] T 01c20eea1b024d7f93f18f0c089efc42 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.507202 1934 tablet_loader.cc:96] loaded metadata for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e])
I20250905 19:52:25.507563 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 399) took 3014 ms. Trace:
I20250905 19:52:25.508764 1934 tablet_loader.cc:96] loaded metadata for tablet 6bb79e3126074736a23b128a8e14fab6 (table default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee])
I20250905 19:52:25.504534 1171 tablet_bootstrap.cc:654] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.508920 1184 rpcz_store.cc:276] 0905 19:52:22.493599 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.511130 (+ 17531us) service_pool.cc:225] Handling call
0905 19:52:25.507411 (+2996281us) ts_tablet_manager.cc:1145] Deleting tablet d296428b70f9411190b6c408d81639f2
0905 19:52:25.507421 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.507534 (+ 113us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2993005}
I20250905 19:52:25.509605 1184 tablet_replica.cc:331] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.509877 1184 raft_consensus.cc:2241] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.510632 1184 raft_consensus.cc:2270] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.511802 1171 tablet_bootstrap.cc:492] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.512066 1171 ts_tablet_manager.cc:1397] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250905 19:52:25.512961 1934 tablet_loader.cc:96] loaded metadata for tablet 6e15350150db413fa4f100171280f46e (table default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff])
I20250905 19:52:25.513624 1184 ts_tablet_manager.cc:1905] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.513854 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 426) took 1986 ms. Trace:
I20250905 19:52:25.514541 1966 tablet.cc:1722] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:25.514657 1185 rpcz_store.cc:276] 0905 19:52:23.527155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.527459 (+ 304us) service_pool.cc:225] Handling call
0905 19:52:25.467000 (+1939541us) ts_tablet_manager.cc:1145] Deleting tablet 01c20eea1b024d7f93f18f0c089efc42
0905 19:52:25.467010 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.483164 (+ 16154us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.503804 (+ 20640us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.513775 (+ 9971us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6170,"tablet-delete.queue_time_us":1599292}
I20250905 19:52:25.516194 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 445) took 1106 ms. Trace:
I20250905 19:52:25.516366 1185 rpcz_store.cc:276] 0905 19:52:24.409659 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.409719 (+ 60us) service_pool.cc:225] Handling call
0905 19:52:25.516003 (+1106284us) ts_tablet_manager.cc:1145] Deleting tablet 00ac56c449ba4b7b8dbd5e87120799d9
0905 19:52:25.516013 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.516136 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1105993}
I20250905 19:52:25.516942 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 447) took 1100 ms. Trace:
I20250905 19:52:25.517380 1185 rpcz_store.cc:276] 0905 19:52:24.416581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.416633 (+ 52us) service_pool.cc:225] Handling call
0905 19:52:25.516805 (+1100172us) ts_tablet_manager.cc:1145] Deleting tablet a1b4989f17454aa58c05727b797c2009
0905 19:52:25.516813 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.516894 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1099944}
I20250905 19:52:25.517930 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 446) took 1101 ms. Trace:
I20250905 19:52:25.518009 1169 tablet_bootstrap.cc:492] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.518187 1169 ts_tablet_manager.cc:1397] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.000s
I20250905 19:52:25.519014 1169 raft_consensus.cc:357] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.519246 1169 raft_consensus.cc:383] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.519346 1169 raft_consensus.cc:738] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.519701 1169 consensus_queue.cc:260] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.518118 1185 rpcz_store.cc:276] 0905 19:52:24.416467 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.416896 (+ 429us) service_pool.cc:225] Handling call
0905 19:52:25.517792 (+1100896us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:25.517800 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.517873 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1100713}
I20250905 19:52:25.520531 1934 tablet_loader.cc:96] loaded metadata for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e])
I20250905 19:52:25.521031 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 448) took 1097 ms. Trace:
I20250905 19:52:25.521224 1934 tablet_loader.cc:96] loaded metadata for tablet 6f3473785fff45d692bd50b1015ffe39 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498])
I20250905 19:52:25.523427 1798 raft_consensus.cc:491] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.523612 1798 raft_consensus.cc:513] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.524719 1934 tablet_loader.cc:96] loaded metadata for tablet 74eb2632a6414e52b9ad67f53c182679 (table default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906])
I20250905 19:52:25.514636 1171 raft_consensus.cc:357] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.526283 1798 leader_election.cc:290] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.525847 1584 raft_consensus.cc:491] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.525935 1658 raft_consensus.cc:491] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.526503 1584 raft_consensus.cc:513] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.526691 1658 raft_consensus.cc:513] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.526921 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:25.527225 1171 raft_consensus.cc:383] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.527336 1171 raft_consensus.cc:738] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:25.527467 986 leader_election.cc:343] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.527572 1171 consensus_queue.cc:260] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.527997 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.528093 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.528162 1171 ts_tablet_manager.cc:1428] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250905 19:52:25.528535 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.528602 984 leader_election.cc:343] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.528715 984 leader_election.cc:304] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.529311 1184 ts_tablet_manager.cc:1918] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:25.529469 1184 log.cc:1199] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/66df0522aa2842f68aa824571fa61b05
I20250905 19:52:25.529656 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.529845 1047 raft_consensus.cc:2466] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:25.529892 1184 ts_tablet_manager.cc:1939] T 66df0522aa2842f68aa824571fa61b05 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.530055 1798 raft_consensus.cc:2747] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.530252 1171 tablet_bootstrap.cc:492] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:25.530426 848 leader_election.cc:343] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.530617 848 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.530707 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 403) took 2929 ms. Trace:
I20250905 19:52:25.530800 1184 rpcz_store.cc:276] 0905 19:52:22.601451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.616093 (+ 14642us) service_pool.cc:225] Handling call
0905 19:52:25.509542 (+2893449us) ts_tablet_manager.cc:1145] Deleting tablet 66df0522aa2842f68aa824571fa61b05
0905 19:52:25.509551 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.519070 (+ 9519us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.529182 (+ 10112us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.530657 (+ 1475us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2892674}
W20250905 19:52:25.531279 849 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2
I20250905 19:52:25.531396 849 leader_election.cc:304] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.531687 849 leader_election.cc:304] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.532008 1672 raft_consensus.cc:2747] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.532152 1171 tablet_bootstrap.cc:654] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.532131 1934 tablet_loader.cc:96] loaded metadata for tablet 7cdd4de055864d4fa6adb36bf190e609 (table default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18])
I20250905 19:52:25.532244 1672 raft_consensus.cc:2802] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.532323 1672 raft_consensus.cc:491] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.532436 1672 raft_consensus.cc:3058] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.532683 1169 ts_tablet_manager.cc:1428] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250905 19:52:25.532719 1934 tablet_loader.cc:96] loaded metadata for tablet 7e8bfe047a6e4cc087eea75244da5b41 (table default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646])
I20250905 19:52:25.533270 1934 tablet_loader.cc:96] loaded metadata for tablet 7ef9c9c6f81c4daea9008ada454b0659 (table default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499])
I20250905 19:52:25.533164 1185 rpcz_store.cc:276] 0905 19:52:24.423061 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.423116 (+ 55us) service_pool.cc:225] Handling call
0905 19:52:25.520879 (+1097763us) ts_tablet_manager.cc:1145] Deleting tablet 19e4edddafcd4d22a0e0ae66b0b953e0
0905 19:52:25.520887 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.520976 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1097526}
I20250905 19:52:25.533624 1934 tablet_loader.cc:96] loaded metadata for tablet 7f34bdf651d1412690039dc6ea1095d0 (table default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967])
I20250905 19:52:25.533622 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 449) took 1095 ms. Trace:
I20250905 19:52:25.533717 1185 rpcz_store.cc:276] 0905 19:52:24.438581 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.438638 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:25.533434 (+1094796us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:25.533443 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.533568 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1094580}
I20250905 19:52:25.534000 1934 tablet_loader.cc:96] loaded metadata for tablet 8092952d755848ccaba3a2ad94b66dad (table default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f])
I20250905 19:52:25.534077 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 451) took 1081 ms. Trace:
I20250905 19:52:25.534171 1185 rpcz_store.cc:276] 0905 19:52:24.453028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.453110 (+ 82us) service_pool.cc:225] Handling call
0905 19:52:25.533943 (+1080833us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:25.533951 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.534026 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1080039}
I20250905 19:52:25.534343 1934 tablet_loader.cc:96] loaded metadata for tablet 813f208881ac4782874138987a710d93 (table default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7])
I20250905 19:52:25.534504 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 452) took 1065 ms. Trace:
I20250905 19:52:25.534585 1185 rpcz_store.cc:276] 0905 19:52:24.469515 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.469586 (+ 71us) service_pool.cc:225] Handling call
0905 19:52:25.534366 (+1064780us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:25.534373 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.534450 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1064549}
I20250905 19:52:25.534915 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 453) took 1063 ms. Trace:
I20250905 19:52:25.534996 1185 rpcz_store.cc:276] 0905 19:52:24.471565 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.471623 (+ 58us) service_pool.cc:225] Handling call
0905 19:52:25.534793 (+1063170us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:25.534800 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.534865 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1062976}
I20250905 19:52:25.535298 1934 tablet_loader.cc:96] loaded metadata for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4])
I20250905 19:52:25.535355 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 450) took 1084 ms. Trace:
I20250905 19:52:25.535420 1672 raft_consensus.cc:513] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.535445 1185 rpcz_store.cc:276] 0905 19:52:24.451188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.482390 (+ 31202us) service_pool.cc:225] Handling call
0905 19:52:25.535186 (+1052796us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:25.535195 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.535299 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1052042}
I20250905 19:52:25.535679 1934 tablet_loader.cc:96] loaded metadata for tablet 862214a31059407baefef8a390aaea69 (table default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482])
I20250905 19:52:25.535715 1185 tablet_replica.cc:331] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.536131 1934 tablet_loader.cc:96] loaded metadata for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7])
I20250905 19:52:25.531244 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 400) took 3036 ms. Trace:
I20250905 19:52:25.538589 1184 rpcz_store.cc:276] 0905 19:52:22.494639 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:22.494681 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:25.531074 (+3036393us) ts_tablet_manager.cc:1145] Deleting tablet fd1280fefc3e43db9928567a734644f1
0905 19:52:25.531083 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.531194 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2910918}
I20250905 19:52:25.538934 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:25.539088 1184 tablet_replica.cc:331] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.539270 1184 raft_consensus.cc:2241] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:25.539458 1584 leader_election.cc:290] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.539621 1184 raft_consensus.cc:2270] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.536998 1934 tablet_loader.cc:96] loaded metadata for tablet 8e0c478efe044ad0867e47f49374a66b (table default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c])
W20250905 19:52:25.539808 848 leader_election.cc:343] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.540789 1934 tablet_loader.cc:96] loaded metadata for tablet 8ee9e38dc3574c1ba9190e06557423f6 (table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e])
I20250905 19:52:25.541101 1169 tablet_bootstrap.cc:492] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.535923 1185 raft_consensus.cc:2241] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:25.541759 1185 pending_rounds.cc:70] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:25.541749 1184 ts_tablet_manager.cc:1905] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.541841 1185 pending_rounds.cc:77] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089566209241088 op_type: NO_OP noop_request { }
I20250905 19:52:25.541966 1185 raft_consensus.cc:2887] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:25.542079 1185 raft_consensus.cc:2270] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.542850 1169 tablet_bootstrap.cc:654] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.543387 1934 tablet_loader.cc:96] loaded metadata for tablet 8faf429d811347a88880e8198daa59cc (table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c])
I20250905 19:52:25.544656 1170 tablet_bootstrap.cc:492] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.544829 1170 ts_tablet_manager.cc:1397] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.004s
I20250905 19:52:25.545230 1934 tablet_loader.cc:96] loaded metadata for tablet 8feaf4ec6b0e4ce9909dd72258912935 (table default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69])
I20250905 19:52:25.527199 1658 leader_election.cc:290] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.545871 1934 tablet_loader.cc:96] loaded metadata for tablet 91396422d0bb468395db04aa80262882 (table default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad])
I20250905 19:52:25.545794 1170 raft_consensus.cc:357] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.545992 1170 raft_consensus.cc:383] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.546095 1170 raft_consensus.cc:738] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.546330 1170 consensus_queue.cc:260] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.546876 1934 tablet_loader.cc:96] loaded metadata for tablet 9193487b4ab74dafa76a386ddcaf674b (table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0])
I20250905 19:52:25.546913 1170 ts_tablet_manager.cc:1428] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:25.548076 1672 leader_election.cc:290] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.548619 1934 tablet_loader.cc:96] loaded metadata for tablet 923ad8e40a07409e8f6a0152a0b2ef75 (table default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6])
I20250905 19:52:25.549338 1934 tablet_loader.cc:96] loaded metadata for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b])
I20250905 19:52:25.550411 1170 tablet_bootstrap.cc:492] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.550872 1972 tablet.cc:1722] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:25.553546 1170 tablet_bootstrap.cc:654] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.557740 1184 ts_tablet_manager.cc:1918] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:25.558492 1934 tablet_loader.cc:96] loaded metadata for tablet 95b32fff7d4745c79185a5d894ef3342 (table default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c])
I20250905 19:52:25.559146 1185 ts_tablet_manager.cc:1905] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.559625 1934 tablet_loader.cc:96] loaded metadata for tablet 9630f1feb6d74472b5f6f4b54151311c (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39])
I20250905 19:52:25.563253 1672 raft_consensus.cc:491] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.563549 1672 raft_consensus.cc:513] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.563884 1170 tablet_bootstrap.cc:492] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.564086 1170 ts_tablet_manager.cc:1397] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.005s
I20250905 19:52:25.564848 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28e403dc2f6471ebea1805fd843cfb1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.564873 1170 raft_consensus.cc:357] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.565127 1170 raft_consensus.cc:383] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.565196 1170 raft_consensus.cc:738] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.565466 1170 consensus_queue.cc:260] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.565555 848 leader_election.cc:343] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.565910 1672 leader_election.cc:290] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.567361 1170 ts_tablet_manager.cc:1428] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:25.570833 1934 tablet_loader.cc:96] loaded metadata for tablet 97d949c3c24e40fe8fefd3a5758d57ea (table default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5])
I20250905 19:52:25.571533 1934 tablet_loader.cc:96] loaded metadata for tablet 98e4e6d274574c0580315dcc2323c3d6 (table default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1])
I20250905 19:52:25.572340 1934 tablet_loader.cc:96] loaded metadata for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3])
I20250905 19:52:25.572686 1170 tablet_bootstrap.cc:492] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.572814 1934 tablet_loader.cc:96] loaded metadata for tablet 9b3acf17eab64913b5b33677196a5781 (table default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d])
I20250905 19:52:25.573459 1934 tablet_loader.cc:96] loaded metadata for tablet 9db56577a26e407b97fc33158c2c466c (table default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7])
I20250905 19:52:25.559031 1169 tablet_bootstrap.cc:492] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.563083 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8feaf4ec6b0e4ce9909dd72258912935" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:25.573841 1050 raft_consensus.cc:3058] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.573674 1184 log.cc:1199] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/0d4204f61929498188c18529148a8d13
I20250905 19:52:25.574559 1184 ts_tablet_manager.cc:1939] T 0d4204f61929498188c18529148a8d13 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.574908 1934 tablet_loader.cc:96] loaded metadata for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda (table default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e])
I20250905 19:52:25.575807 1934 tablet_loader.cc:96] loaded metadata for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac])
I20250905 19:52:25.576768 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 419) took 2246 ms. Trace:
I20250905 19:52:25.576803 1849 raft_consensus.cc:491] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.576977 1849 raft_consensus.cc:513] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.577423 1934 tablet_loader.cc:96] loaded metadata for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77])
I20250905 19:52:25.577675 1849 leader_election.cc:290] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.577570 1184 rpcz_store.cc:276] 0905 19:52:23.330531 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.330597 (+ 66us) service_pool.cc:225] Handling call
0905 19:52:25.538986 (+2208389us) ts_tablet_manager.cc:1145] Deleting tablet 0d4204f61929498188c18529148a8d13
0905 19:52:25.538996 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.549800 (+ 10804us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.557576 (+ 7776us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.576730 (+ 19154us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2208123}
I20250905 19:52:25.578824 1934 tablet_loader.cc:96] loaded metadata for tablet a311f9b943d143a9a52d7771ce7186d7 (table default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4])
I20250905 19:52:25.579182 1050 raft_consensus.cc:2466] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.579152 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.579505 1169 ts_tablet_manager.cc:1397] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.002s
I20250905 19:52:25.579545 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 418) took 2249 ms. Trace:
I20250905 19:52:25.579785 1184 rpcz_store.cc:276] 0905 19:52:23.330405 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.330589 (+ 184us) service_pool.cc:225] Handling call
0905 19:52:25.579198 (+2248609us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:25.579209 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.579505 (+ 296us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2248212}
I20250905 19:52:25.580147 1672 raft_consensus.cc:491] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.580364 1672 raft_consensus.cc:513] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.580451 1169 raft_consensus.cc:357] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.580641 1169 raft_consensus.cc:383] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.580730 1169 raft_consensus.cc:738] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.580453 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 417) took 2250 ms. Trace:
I20250905 19:52:25.581456 1185 ts_tablet_manager.cc:1918] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:25.582295 1934 tablet_loader.cc:96] loaded metadata for tablet a3a50526bb604a218649b24cbf65a65e (table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716])
I20250905 19:52:25.582346 1170 tablet_bootstrap.cc:654] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.582484 1184 rpcz_store.cc:276] 0905 19:52:23.330291 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.352848 (+ 22557us) service_pool.cc:225] Handling call
0905 19:52:25.580255 (+2227407us) ts_tablet_manager.cc:1145] Deleting tablet 3609de83732d4ad8857372acfbb5c32a
0905 19:52:25.580320 (+ 65us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.580410 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2188089}
I20250905 19:52:25.582545 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.583178 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 420) took 2069 ms. Trace:
I20250905 19:52:25.583293 1184 rpcz_store.cc:276] 0905 19:52:23.513538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.513621 (+ 83us) service_pool.cc:225] Handling call
0905 19:52:25.583035 (+2069414us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:25.583044 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.583141 (+ 97us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2066319}
I20250905 19:52:25.583779 1185 log.cc:1199] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:25.580971 1169 consensus_queue.cc:260] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.584942 511 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.62:38687 timed out after 0.533s (SENT) [suppressed 4 similar messages]
I20250905 19:52:25.588752 1169 ts_tablet_manager.cc:1428] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250905 19:52:25.589174 1169 tablet_bootstrap.cc:492] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:25.583925 848 leader_election.cc:343] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.585690 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2466a1bafc8347e483ca5e28d552bfc9" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:25.586313 849 leader_election.cc:343] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:25.593621 849 leader_election.cc:304] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.594092 849 leader_election.cc:304] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.594444 1973 raft_consensus.cc:2747] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.587700 1171 tablet_bootstrap.cc:492] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.594679 1973 raft_consensus.cc:2802] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.594676 712 leader_election.cc:304] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.594834 1973 raft_consensus.cc:695] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.595203 1849 raft_consensus.cc:2747] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.595105 1973 consensus_queue.cc:237] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.595753 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.596005 1047 raft_consensus.cc:2466] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:25.594830 1171 ts_tablet_manager.cc:1397] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.065s user 0.012s sys 0.006s
I20250905 19:52:25.596540 849 leader_election.cc:304] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.596840 1584 raft_consensus.cc:2802] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.596946 1584 raft_consensus.cc:491] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.597031 1584 raft_consensus.cc:3058] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.597600 1171 raft_consensus.cc:357] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.597833 1171 raft_consensus.cc:383] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.597954 1171 raft_consensus.cc:738] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.598217 1171 consensus_queue.cc:260] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.598981 1171 ts_tablet_manager.cc:1428] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:25.599321 1171 tablet_bootstrap.cc:492] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.599745 1584 raft_consensus.cc:513] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.600272 1584 leader_election.cc:290] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.594120 1672 leader_election.cc:290] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.600930 770 raft_consensus.cc:3058] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.601919 1672 raft_consensus.cc:491] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.602069 1672 raft_consensus.cc:513] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.602300 1171 tablet_bootstrap.cc:654] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.602656 1672 leader_election.cc:290] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.604138 1185 ts_tablet_manager.cc:1939] T 4fe94d1254d04949aa25346d8e323c3b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.604215 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:25.604591 849 leader_election.cc:343] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:25.604627 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5f076d9917f40d899d7422ad974fe0f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.604748 848 leader_election.cc:343] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.604776 1798 raft_consensus.cc:491] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.604916 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 455) took 1106 ms. Trace:
I20250905 19:52:25.604938 1798 raft_consensus.cc:513] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.605028 1185 rpcz_store.cc:276] 0905 19:52:24.498370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.498921 (+ 551us) service_pool.cc:225] Handling call
0905 19:52:25.535634 (+1036713us) ts_tablet_manager.cc:1145] Deleting tablet 4fe94d1254d04949aa25346d8e323c3b
0905 19:52:25.535642 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.562920 (+ 27278us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.581304 (+ 18384us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.604863 (+ 23559us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1036393}
I20250905 19:52:25.605589 1798 leader_election.cc:290] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.605923 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9630f1feb6d74472b5f6f4b54151311c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.606351 1934 tablet_loader.cc:96] loaded metadata for tablet a551e641a19f4d3f924fb176c4a25ca7 (table default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba])
I20250905 19:52:25.606482 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9630f1feb6d74472b5f6f4b54151311c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:25.607023 984 leader_election.cc:343] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.607685 1934 tablet_loader.cc:96] loaded metadata for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913])
I20250905 19:52:25.608309 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 422) took 2080 ms. Trace:
W20250905 19:52:25.608395 848 leader_election.cc:343] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.608510 848 leader_election.cc:304] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.608457 1184 rpcz_store.cc:276] 0905 19:52:23.527892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.527954 (+ 62us) service_pool.cc:225] Handling call
0905 19:52:25.608147 (+2080193us) ts_tablet_manager.cc:1145] Deleting tablet 19e4edddafcd4d22a0e0ae66b0b953e0
0905 19:52:25.608163 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.608264 (+ 101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1745132}
W20250905 19:52:25.609112 986 leader_election.cc:343] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.609238 986 leader_election.cc:304] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.609623 1798 raft_consensus.cc:2747] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.609910 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 421) took 2082 ms. Trace:
I20250905 19:52:25.610061 1184 rpcz_store.cc:276] 0905 19:52:23.527396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:23.527593 (+ 197us) service_pool.cc:225] Handling call
0905 19:52:25.609730 (+2082137us) ts_tablet_manager.cc:1145] Deleting tablet a2ff206366ad4344b4df5fc6a5584f8a
0905 19:52:25.609744 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.609869 (+ 125us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1736404}
I20250905 19:52:25.610426 1184 tablet_replica.cc:331] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:25.610657 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:25.610713 1184 raft_consensus.cc:2241] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.610874 1047 raft_consensus.cc:3058] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.610929 1184 raft_consensus.cc:2270] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.612222 1185 tablet_replica.cc:331] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.612500 1185 raft_consensus.cc:2241] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.612728 1185 raft_consensus.cc:2270] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.613170 1047 raft_consensus.cc:2466] T 7cdd4de055864d4fa6adb36bf190e609 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.613476 1185 ts_tablet_manager.cc:1905] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.614923 1184 ts_tablet_manager.cc:1905] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.617728 1934 tablet_loader.cc:96] loaded metadata for tablet ac2e51a9071c4e1dae9e147687291175 (table default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef])
I20250905 19:52:25.613628 849 leader_election.cc:304] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.622244 1672 raft_consensus.cc:2802] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.622409 1672 raft_consensus.cc:695] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.622694 1672 consensus_queue.cc:237] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.616765 1973 raft_consensus.cc:2747] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.615828 1169 tablet_bootstrap.cc:654] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.617780 1798 raft_consensus.cc:491] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.624197 1798 raft_consensus.cc:513] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.625093 1798 leader_election.cc:290] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.625857 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.626549 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:25.627110 984 leader_election.cc:343] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.627589 986 leader_election.cc:343] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.627696 986 leader_election.cc:304] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.628144 1798 raft_consensus.cc:2747] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.629096 1934 tablet_loader.cc:96] loaded metadata for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf])
I20250905 19:52:25.629177 1169 tablet_bootstrap.cc:492] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.629367 1169 ts_tablet_manager.cc:1397] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.001s
I20250905 19:52:25.630295 1169 raft_consensus.cc:357] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.630525 1169 raft_consensus.cc:383] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.630604 1169 raft_consensus.cc:738] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.631196 1934 tablet_loader.cc:96] loaded metadata for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e])
I20250905 19:52:25.633057 1169 consensus_queue.cc:260] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.634809 986 connection.cc:537] server connection from 127.0.0.1:44176 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:25.634840 847 connection.cc:537] server connection from 127.0.0.1:51212 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:25.635572 1472 connection.cc:537] client connection to 127.0.104.62:38687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:25.635587 712 connection.cc:537] server connection from 127.0.0.1:48712 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:25.635823 1472 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:25.636375 1171 tablet_bootstrap.cc:492] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.636566 1171 ts_tablet_manager.cc:1397] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.008s
I20250905 19:52:25.637331 1171 raft_consensus.cc:357] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.637549 1171 raft_consensus.cc:383] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.637624 1171 raft_consensus.cc:738] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.637883 1171 consensus_queue.cc:260] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.638440 1169 ts_tablet_manager.cc:1428] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250905 19:52:25.638809 1171 ts_tablet_manager.cc:1428] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.639173 1171 tablet_bootstrap.cc:492] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:25.642449 1109 connection.cc:537] client connection to 127.0.104.62:38687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:25.643404 1169 tablet_bootstrap.cc:492] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:25.643718 1109 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.62:38687)
I20250905 19:52:25.644301 1934 tablet_loader.cc:96] loaded metadata for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd])
W20250905 19:52:25.644649 1109 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:38687)
I20250905 19:52:25.644793 1934 tablet_loader.cc:96] loaded metadata for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75])
I20250905 19:52:25.645185 1169 tablet_bootstrap.cc:654] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:25.645252 1109 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.62:38687)
W20250905 19:52:25.646004 1109 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.62:38687)
I20250905 19:52:25.646242 1934 tablet_loader.cc:96] loaded metadata for tablet b1f46261822442e0a961d2d63bdb35c8 (table default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801])
I20250905 19:52:25.646696 1934 tablet_loader.cc:96] loaded metadata for tablet b2c7906a718c46d9a76587a70d96a1f9 (table default.Table_4447fb37960e42dfa2c6eb8ced9c6b68 [id=99384aac4e624d6db630b249c3ba2543])
I20250905 19:52:25.647578 1973 raft_consensus.cc:491] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.647779 1973 raft_consensus.cc:513] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.648475 1973 leader_election.cc:290] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.649279 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0702063ec1c4f6fba58e6d4f3912212" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.650110 849 leader_election.cc:343] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:25.650460 1171 tablet_bootstrap.cc:654] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.650483 1184 ts_tablet_manager.cc:1918] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250905 19:52:25.650548 848 leader_election.cc:343] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.650663 1184 log.cc:1199] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/00ac56c449ba4b7b8dbd5e87120799d9
I20250905 19:52:25.650656 848 leader_election.cc:304] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.651121 1184 ts_tablet_manager.cc:1939] T 00ac56c449ba4b7b8dbd5e87120799d9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:25.651952 1973 raft_consensus.cc:2747] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.656456 1934 tablet_loader.cc:96] loaded metadata for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0])
I20250905 19:52:25.657619 1170 tablet_bootstrap.cc:492] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.657819 1170 ts_tablet_manager.cc:1397] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.085s user 0.008s sys 0.029s
I20250905 19:52:25.658914 1170 raft_consensus.cc:357] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.659162 1170 raft_consensus.cc:383] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.659269 1170 raft_consensus.cc:738] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.659571 1170 consensus_queue.cc:260] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.661178 1170 ts_tablet_manager.cc:1428] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:25.661784 1170 tablet_bootstrap.cc:492] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.663604 1849 raft_consensus.cc:491] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.663786 1849 raft_consensus.cc:513] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.664184 1170 tablet_bootstrap.cc:654] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.664466 1849 leader_election.cc:290] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.666631 1934 tablet_loader.cc:96] loaded metadata for tablet be1315824fd749049a62af3e6a113b9a (table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f])
I20250905 19:52:25.666431 1672 raft_consensus.cc:491] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.667171 1672 raft_consensus.cc:513] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.667274 1934 tablet_loader.cc:96] loaded metadata for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422])
I20250905 19:52:25.667665 1170 tablet_bootstrap.cc:492] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.667827 1170 ts_tablet_manager.cc:1397] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.003s
I20250905 19:52:25.668054 1672 leader_election.cc:290] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.668476 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.668711 1170 raft_consensus.cc:357] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.669157 848 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.668952 1170 raft_consensus.cc:383] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.669382 1170 raft_consensus.cc:738] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.669720 1170 consensus_queue.cc:260] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.669781 712 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 4c76077e4adf45a7adc736b0e69032eb
W20250905 19:52:25.670122 711 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:25.670289 711 leader_election.cc:304] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.670343 1170 ts_tablet_manager.cc:1428] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.670693 1170 tablet_bootstrap.cc:492] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.670871 1849 raft_consensus.cc:2747] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.673209 1170 tablet_bootstrap.cc:654] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.675313 1934 tablet_loader.cc:96] loaded metadata for tablet c14dc571b586405f8fd1eaa452998978 (table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0])
W20250905 19:52:25.675947 849 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:25.676178 849 leader_election.cc:304] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.676549 1185 ts_tablet_manager.cc:1918] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.676641 1672 raft_consensus.cc:2747] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.677182 1169 tablet_bootstrap.cc:492] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.677338 1169 ts_tablet_manager.cc:1397] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.020s
I20250905 19:52:25.678308 1169 raft_consensus.cc:357] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.678544 1169 raft_consensus.cc:383] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.678648 1169 raft_consensus.cc:738] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.677389 1185 log.cc:1199] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/262bcd1cfc994bb58e999b141b9bc7ea
I20250905 19:52:25.678910 1169 consensus_queue.cc:260] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.679752 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 440) took 1270 ms. Trace:
I20250905 19:52:25.680160 1184 rpcz_store.cc:276] 0905 19:52:24.405342 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.405402 (+ 60us) service_pool.cc:225] Handling call
0905 19:52:25.610354 (+1204952us) ts_tablet_manager.cc:1145] Deleting tablet 00ac56c449ba4b7b8dbd5e87120799d9
0905 19:52:25.610366 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.641109 (+ 30743us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.650318 (+ 9209us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.676084 (+ 25766us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1204691}
I20250905 19:52:25.680868 1185 ts_tablet_manager.cc:1939] T 262bcd1cfc994bb58e999b141b9bc7ea P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:25.681418 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 440) could send its response
I20250905 19:52:25.682724 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 456) took 1184 ms. Trace:
I20250905 19:52:25.683773 1169 ts_tablet_manager.cc:1428] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
W20250905 19:52:25.684386 710 connection.cc:537] client connection to 127.0.104.62:38687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:25.685413 511 connection.cc:537] client connection to 127.0.104.62:38687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:25.685464 1169 tablet_bootstrap.cc:492] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:25.685566 511 leader_election.cc:336] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:25.685699 984 connection.cc:537] client connection to 127.0.104.62:38687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:25.686440 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 441) took 1242 ms. Trace:
I20250905 19:52:25.690481 1169 tablet_bootstrap.cc:654] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.696841 1934 tablet_loader.cc:96] loaded metadata for tablet c4b31443b0f840749e12435e197a204a (table default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f])
I20250905 19:52:25.698262 1934 tablet_loader.cc:96] loaded metadata for tablet c69a41cee5ff4f8f9bccaa513c096118 (table default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d])
I20250905 19:52:25.686554 1184 rpcz_store.cc:276] 0905 19:52:24.443953 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.444282 (+ 329us) service_pool.cc:225] Handling call
0905 19:52:25.686236 (+1241954us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:25.686247 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.686382 (+ 135us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1238567}
W20250905 19:52:25.698781 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 441) could send its response
I20250905 19:52:25.699518 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 443) took 1255 ms. Trace:
W20250905 19:52:25.684291 848 connection.cc:537] client connection to 127.0.104.62:38687 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:25.700711 1185 rpcz_store.cc:276] 0905 19:52:24.498589 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.499289 (+ 700us) service_pool.cc:225] Handling call
0905 19:52:25.612116 (+1112827us) ts_tablet_manager.cc:1145] Deleting tablet 262bcd1cfc994bb58e999b141b9bc7ea
0905 19:52:25.612130 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.652536 (+ 40406us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.669229 (+ 16693us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.682675 (+ 13446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1112630}
I20250905 19:52:25.701444 1798 raft_consensus.cc:491] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.701632 1798 raft_consensus.cc:513] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.702282 1798 leader_election.cc:290] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.708503 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4edfa9a656554e02b6649e8430c215fb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.709744 984 leader_election.cc:343] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.710734 1934 tablet_loader.cc:96] loaded metadata for tablet c77dd068ea5c495ca8aa0e7a6823d792 (table default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d])
I20250905 19:52:25.701166 1184 rpcz_store.cc:276] 0905 19:52:24.444223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.444746 (+ 523us) service_pool.cc:225] Handling call
0905 19:52:25.699300 (+1254554us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:25.699314 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.699458 (+ 144us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1254300}
W20250905 19:52:25.712342 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 443) could send its response
W20250905 19:52:25.712688 847 connection.cc:441] server connection from 127.0.0.1:51212 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 456) could send its response
I20250905 19:52:25.713235 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 442) took 1269 ms. Trace:
I20250905 19:52:25.713547 1185 tablet_replica.cc:331] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.713884 1185 raft_consensus.cc:2241] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.714033 1184 rpcz_store.cc:276] 0905 19:52:24.444123 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.479900 (+ 35777us) service_pool.cc:225] Handling call
0905 19:52:25.713018 (+1233118us) ts_tablet_manager.cc:1145] Deleting tablet a1b4989f17454aa58c05727b797c2009
0905 19:52:25.713033 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.713166 (+ 133us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090093}
I20250905 19:52:25.715371 1934 tablet_loader.cc:96] loaded metadata for tablet c870e689af254fb0acb15330577e5c42 (table default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52])
I20250905 19:52:25.695188 1171 tablet_bootstrap.cc:492] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.702929 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4edfa9a656554e02b6649e8430c215fb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:25.716180 913 raft_consensus.cc:2466] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
W20250905 19:52:25.716204 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 442) could send its response
I20250905 19:52:25.716511 1185 raft_consensus.cc:2270] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.716708 986 leader_election.cc:304] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.717151 1798 raft_consensus.cc:2802] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.717234 1798 raft_consensus.cc:491] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.717257 1185 ts_tablet_manager.cc:1905] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.717331 1798 raft_consensus.cc:3058] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.716225 1171 ts_tablet_manager.cc:1397] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.077s user 0.001s sys 0.012s
I20250905 19:52:25.720239 1798 raft_consensus.cc:513] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.720414 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 444) took 1091 ms. Trace:
I20250905 19:52:25.720536 1184 rpcz_store.cc:276] 0905 19:52:24.628614 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.629135 (+ 521us) service_pool.cc:225] Handling call
0905 19:52:25.720180 (+1091045us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:25.720198 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.720337 (+ 139us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1090753}
W20250905 19:52:25.720903 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 444) could send its response
I20250905 19:52:25.720844 1171 raft_consensus.cc:357] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.720994 1798 leader_election.cc:290] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.721030 1171 raft_consensus.cc:383] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.721113 1171 raft_consensus.cc:738] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.722150 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4edfa9a656554e02b6649e8430c215fb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:25.722324 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 446) took 1093 ms. Trace:
I20250905 19:52:25.722430 1184 rpcz_store.cc:276] 0905 19:52:24.628841 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.629455 (+ 614us) service_pool.cc:225] Handling call
0905 19:52:25.722108 (+1092653us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:25.722122 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.722271 (+ 149us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1092472}
W20250905 19:52:25.722695 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 446) could send its response
W20250905 19:52:25.723009 984 leader_election.cc:343] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.721875 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4edfa9a656554e02b6649e8430c215fb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:25.724439 913 raft_consensus.cc:3058] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:25.724916 1989 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 19:52:25.654631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38687 (local address 127.0.0.1:39866)
0905 19:52:25.701550 (+ 46919us) negotiation.cc:107] Waiting for socket to connect
0905 19:52:25.701757 (+ 207us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":46777,"threads_started":1}
I20250905 19:52:25.724444 1171 consensus_queue.cc:260] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.726148 1171 ts_tablet_manager.cc:1428] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:25.726716 1171 tablet_bootstrap.cc:492] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.728065 1798 raft_consensus.cc:491] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.728224 1798 raft_consensus.cc:513] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.728681 1171 tablet_bootstrap.cc:654] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.728897 1798 leader_election.cc:290] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.731032 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e9994863c374354838aa0f817d0057e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:25.729595 913 raft_consensus.cc:2466] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:25.731842 986 leader_election.cc:343] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.732072 986 leader_election.cc:304] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.732520 1798 raft_consensus.cc:2802] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.733364 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e9994863c374354838aa0f817d0057e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.733976 984 leader_election.cc:343] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.734113 984 leader_election.cc:304] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:25.734514 1472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38687, user_credentials={real_user=slave}} blocked reactor thread for 98725.6us
I20250905 19:52:25.735584 1934 tablet_loader.cc:96] loaded metadata for tablet c8e53152ada84954b12bbfe9d1fe3c16 (table default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92])
I20250905 19:52:25.738703 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 445) took 1102 ms. Trace:
I20250905 19:52:25.738871 1184 rpcz_store.cc:276] 0905 19:52:24.628764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.630550 (+ 1786us) service_pool.cc:225] Handling call
0905 19:52:25.731454 (+1100904us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:25.731469 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.731611 (+ 142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1089261}
W20250905 19:52:25.739220 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 445) could send its response
I20250905 19:52:25.739858 1798 raft_consensus.cc:695] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:25.740293 1999 raft_consensus.cc:2747] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.740216 1798 consensus_queue.cc:237] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.741439 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 71315.5us
I20250905 19:52:25.742790 1184 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 447) took 1113 ms. Trace:
I20250905 19:52:25.742911 1184 rpcz_store.cc:276] 0905 19:52:24.628915 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.630674 (+ 1759us) service_pool.cc:225] Handling call
0905 19:52:25.742576 (+1111902us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:25.742594 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.742720 (+ 126us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1087183}
W20250905 19:52:25.743220 986 connection.cc:441] server connection from 127.0.0.1:44176 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:44176 (request call id 447) could send its response
I20250905 19:52:25.744580 1934 tablet_loader.cc:96] loaded metadata for tablet ca646f7c162e41469f283657e72823b4 (table default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9])
I20250905 19:52:25.744722 1169 tablet_bootstrap.cc:492] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.744891 1169 ts_tablet_manager.cc:1397] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.059s user 0.004s sys 0.024s
W20250905 19:52:25.745162 1472 leader_election.cc:336] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection refused (error 111)
W20250905 19:52:25.745711 1472 consensus_peers.cc:489] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e -> Peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Couldn't send request to peer c401c9f3020b4ae5931c566c6bcb1711. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:25.745750 1169 raft_consensus.cc:357] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.745860 1472 leader_election.cc:336] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection refused (error 111)
I20250905 19:52:25.745955 1169 raft_consensus.cc:383] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.746038 1169 raft_consensus.cc:738] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.746304 1169 consensus_queue.cc:260] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.746721 534 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Refusing update from remote peer e38333af29db4bf6af41ee5a25d6701e: Log matching property violated. Preceding OpId in replica: term: 2 index: 593. Preceding OpId from leader: term: 3 index: 594. (index mismatch)
I20250905 19:52:25.746809 1169 ts_tablet_manager.cc:1428] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:25.747128 1169 tablet_bootstrap.cc:492] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.747637 1930 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 594, Last known committed idx: 593, Time since last communication: 0.000s
I20250905 19:52:25.751021 2002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38333af29db4bf6af41ee5a25d6701e. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:25.751358 2002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:25.751888 1672 raft_consensus.cc:491] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.752079 1672 raft_consensus.cc:513] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.752638 1672 leader_election.cc:290] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.753242 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.753432 1047 raft_consensus.cc:2466] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:25.754179 849 leader_election.cc:304] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:25.754532 1672 raft_consensus.cc:2802] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.754608 1672 raft_consensus.cc:491] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.754688 1672 raft_consensus.cc:3058] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.755206 1169 tablet_bootstrap.cc:654] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.757834 1672 raft_consensus.cc:513] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.758985 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:25.759160 1047 raft_consensus.cc:3058] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.760619 1171 tablet_bootstrap.cc:492] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.760802 1171 ts_tablet_manager.cc:1397] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250905 19:52:25.761765 1171 raft_consensus.cc:357] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.761988 1171 raft_consensus.cc:383] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.762063 1047 raft_consensus.cc:2466] T 0039b8f0ba704907b2820c9e3a69cd73 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.762084 1171 raft_consensus.cc:738] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.751704 1934 tablet_loader.cc:96] loaded metadata for tablet cd5312852fe345d98039aeea241b7fe8 (table default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d])
I20250905 19:52:25.762679 849 leader_election.cc:304] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:25.762955 1973 raft_consensus.cc:2802] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.763064 1672 leader_election.cc:290] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.763142 1973 raft_consensus.cc:695] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.763437 1973 consensus_queue.cc:237] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.764120 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.764667 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:25.765208 848 leader_election.cc:343] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.765689 848 leader_election.cc:343] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.767215 1991 raft_consensus.cc:491] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.767407 1991 raft_consensus.cc:513] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.768273 1849 raft_consensus.cc:491] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.768440 1849 raft_consensus.cc:513] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.769024 1849 leader_election.cc:290] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:25.769415 712 leader_election.cc:343] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:25.769811 1849 raft_consensus.cc:491] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.769949 1849 raft_consensus.cc:513] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.770035 712 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:25.770211 1991 leader_election.cc:290] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.770505 1849 leader_election.cc:290] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:25.771034 712 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:25.772753 1584 raft_consensus.cc:491] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.772889 1584 raft_consensus.cc:513] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.773478 1584 leader_election.cc:290] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.774257 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ec559d0f83445892f8285f9bdeb2c8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.774303 1934 tablet_loader.cc:96] loaded metadata for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a])
I20250905 19:52:25.774545 1047 raft_consensus.cc:2466] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:25.775094 849 leader_election.cc:304] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:25.775401 1584 raft_consensus.cc:2802] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.775506 1584 raft_consensus.cc:491] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.775576 1584 raft_consensus.cc:3058] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.776571 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ec559d0f83445892f8285f9bdeb2c8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.778388 1584 raft_consensus.cc:513] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.778981 1934 tablet_loader.cc:96] loaded metadata for tablet d1a11777cf2d4ea9a5cdf792f73aff76 (table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d])
I20250905 19:52:25.779196 1584 leader_election.cc:290] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.779896 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ec559d0f83445892f8285f9bdeb2c8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:25.780109 1050 raft_consensus.cc:3058] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.782867 1050 raft_consensus.cc:2466] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.783388 849 leader_election.cc:304] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:25.784030 848 leader_election.cc:343] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.784469 1584 raft_consensus.cc:2802] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.784678 1658 raft_consensus.cc:491] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.784844 1171 consensus_queue.cc:260] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.784865 1658 raft_consensus.cc:513] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.785192 1170 tablet_bootstrap.cc:492] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.785377 1170 ts_tablet_manager.cc:1397] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.115s user 0.012s sys 0.045s
I20250905 19:52:25.785487 1658 leader_election.cc:290] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.785988 1584 raft_consensus.cc:695] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.786262 1170 raft_consensus.cc:357] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.786324 1584 consensus_queue.cc:237] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.786473 1170 raft_consensus.cc:383] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.786562 1170 raft_consensus.cc:738] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.786813 1170 consensus_queue.cc:260] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.787801 1171 ts_tablet_manager.cc:1428] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.027s user 0.001s sys 0.001s
I20250905 19:52:25.788899 1170 ts_tablet_manager.cc:1428] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:25.789213 1170 tablet_bootstrap.cc:492] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.790966 1170 tablet_bootstrap.cc:654] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.793097 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.793402 1047 raft_consensus.cc:2466] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:25.793918 849 leader_election.cc:304] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:25.795454 1997 raft_consensus.cc:491] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.795504 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ec559d0f83445892f8285f9bdeb2c8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:25.795643 770 raft_consensus.cc:3058] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.796095 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.796594 848 leader_election.cc:343] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.796602 1997 raft_consensus.cc:513] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:25.796919 848 leader_election.cc:343] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.797616 1672 raft_consensus.cc:2802] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.797727 1672 raft_consensus.cc:491] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.797811 1672 raft_consensus.cc:3058] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.798040 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "415508e72ee34bc391c4369b5f8a2f48" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:25.798883 986 leader_election.cc:343] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.800240 1997 leader_election.cc:290] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.800520 1672 raft_consensus.cc:513] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.804028 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "415508e72ee34bc391c4369b5f8a2f48" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.804741 984 leader_election.cc:343] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.804780 1171 tablet_bootstrap.cc:492] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.804955 984 leader_election.cc:304] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.805451 1999 raft_consensus.cc:2747] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:25.806097 711 leader_election.cc:343] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:25.806310 711 leader_election.cc:304] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.806623 1849 raft_consensus.cc:2747] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:25.806803 711 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:25.806933 711 leader_election.cc:304] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.807163 1849 raft_consensus.cc:2747] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:25.807308 711 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:25.807420 711 leader_election.cc:304] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.807662 1849 raft_consensus.cc:2747] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.808096 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:25.809095 848 leader_election.cc:343] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.810410 1934 tablet_loader.cc:96] loaded metadata for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_ad054f6da70449ecad529f69c6aa9705 [id=c31b7e937b6d4adfb299ae8e46330d55])
I20250905 19:52:25.811897 1934 tablet_loader.cc:96] loaded metadata for tablet d59da39e7bcf497a93fb9174328c1814 (table default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1])
I20250905 19:52:25.813529 1672 leader_election.cc:290] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.813809 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e89d23c4cb64f568b25f92a278b9d5e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:25.814103 1047 raft_consensus.cc:3058] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.814450 2002 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:25.814621 2002 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:25.817427 1047 raft_consensus.cc:2466] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.817878 849 leader_election.cc:304] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.819301 1934 tablet_loader.cc:96] loaded metadata for tablet d6b44c00d3b54af6b0752575e6898839 (table default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c])
I20250905 19:52:25.819612 2007 raft_consensus.cc:2802] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.819752 1934 tablet_loader.cc:96] loaded metadata for tablet d709e2b4f10149a1818c1e1431b95937 (table default.Table_19682dbce0bc4480b82d39235ba7f55b [id=bff29fe995d34f32ad9780ad5591a8b2])
I20250905 19:52:25.820650 1185 ts_tablet_manager.cc:1918] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.821023 2007 raft_consensus.cc:695] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.820953 1171 tablet_bootstrap.cc:654] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.821638 1997 raft_consensus.cc:491] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.821848 1997 raft_consensus.cc:513] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.822757 1997 leader_election.cc:290] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.823647 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.824425 984 leader_election.cc:343] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.825261 1943 tablet.cc:1722] T 2466a1bafc8347e483ca5e28d552bfc9 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:25.825961 2004 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:25.826311 2004 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:25.821111 1185 log.cc:1199] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:25.826974 1185 ts_tablet_manager.cc:1939] T 68f2ccd8a7a64b89b44f2d6767ec3466 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.828105 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 457) took 1329 ms. Trace:
I20250905 19:52:25.828300 1185 rpcz_store.cc:276] 0905 19:52:24.498699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.499497 (+ 798us) service_pool.cc:225] Handling call
0905 19:52:25.713457 (+1213960us) ts_tablet_manager.cc:1145] Deleting tablet 68f2ccd8a7a64b89b44f2d6767ec3466
0905 19:52:25.713471 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.776226 (+ 62755us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.783947 (+ 7721us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.827992 (+ 44045us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1213770}
W20250905 19:52:25.828722 847 connection.cc:441] server connection from 127.0.0.1:51212 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 457) could send its response
I20250905 19:52:25.835249 1185 tablet_replica.cc:331] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:25.835641 1185 raft_consensus.cc:2241] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:25.836127 1185 raft_consensus.cc:2270] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:25.836874 1185 ts_tablet_manager.cc:1905] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:25.838289 1169 tablet_bootstrap.cc:492] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.838454 1169 ts_tablet_manager.cc:1397] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.091s user 0.007s sys 0.026s
I20250905 19:52:25.825203 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:25.839064 986 leader_election.cc:343] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.839221 986 leader_election.cc:304] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.839345 1169 raft_consensus.cc:357] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.839520 1169 raft_consensus.cc:383] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.839612 1169 raft_consensus.cc:738] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.839823 1169 consensus_queue.cc:260] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.839995 1999 raft_consensus.cc:2747] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.840399 1169 ts_tablet_manager.cc:1428] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.840687 1169 tablet_bootstrap.cc:492] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.821364 2007 consensus_queue.cc:237] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.843369 1934 tablet_loader.cc:96] loaded metadata for tablet d7155cf5abba4e2cbdce583406f94fad (table default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816])
I20250905 19:52:25.844341 1169 tablet_bootstrap.cc:654] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.849581 1934 tablet_loader.cc:96] loaded metadata for tablet db098a883a61474fb06cb166041ee3e4 (table default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2])
I20250905 19:52:25.850509 1934 tablet_loader.cc:96] loaded metadata for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (table default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786])
I20250905 19:52:25.851306 1934 tablet_loader.cc:96] loaded metadata for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_849b4e364b3f4ff2b47a2347f84d6b0d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef])
I20250905 19:52:25.852367 1934 tablet_loader.cc:96] loaded metadata for tablet dcf448f9abc348df8d2d2501420d8f45 (table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727])
I20250905 19:52:25.852676 1170 tablet_bootstrap.cc:492] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.852869 1170 ts_tablet_manager.cc:1397] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.064s user 0.009s sys 0.013s
I20250905 19:52:25.853014 1934 tablet_loader.cc:96] loaded metadata for tablet de19c897d3314fecb17cedb3370eed93 (table default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb])
I20250905 19:52:25.853437 1934 tablet_loader.cc:96] loaded metadata for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b])
I20250905 19:52:25.853917 1170 raft_consensus.cc:357] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.854156 1170 raft_consensus.cc:383] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.854239 1170 raft_consensus.cc:738] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.854548 1170 consensus_queue.cc:260] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.855080 1171 tablet_bootstrap.cc:492] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.855280 1171 ts_tablet_manager.cc:1397] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.000s
I20250905 19:52:25.855891 2007 raft_consensus.cc:491] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.856726 2007 raft_consensus.cc:513] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.859366 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.859375 2007 leader_election.cc:290] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.856495 1171 raft_consensus.cc:357] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.859560 1171 raft_consensus.cc:383] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.859650 1171 raft_consensus.cc:738] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.859756 1170 ts_tablet_manager.cc:1428] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250905 19:52:25.859915 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:25.859905 1171 consensus_queue.cc:260] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.860178 1050 raft_consensus.cc:2466] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:25.860164 848 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.862685 1170 tablet_bootstrap.cc:492] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.864784 1170 tablet_bootstrap.cc:654] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.860812 849 leader_election.cc:304] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.866550 1672 raft_consensus.cc:2802] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:25.866670 1672 raft_consensus.cc:491] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:25.866797 1672 raft_consensus.cc:3058] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.868342 1169 tablet_bootstrap.cc:492] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.868512 1169 ts_tablet_manager.cc:1397] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.002s
I20250905 19:52:25.869306 1169 raft_consensus.cc:357] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.869745 1169 raft_consensus.cc:383] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.869860 1169 raft_consensus.cc:738] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.869915 1672 raft_consensus.cc:513] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.870162 1169 consensus_queue.cc:260] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.870677 1169 ts_tablet_manager.cc:1428] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.870919 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:25.870990 1672 leader_election.cc:290] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:25.871425 848 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.863750 1584 raft_consensus.cc:491] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.873061 1584 raft_consensus.cc:513] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.873646 1584 leader_election.cc:290] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.873979 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2ed1bc09c94526bc9662f2d4d21f8f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.874517 849 leader_election.cc:343] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:25.874569 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
W20250905 19:52:25.874677 848 leader_election.cc:343] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.874739 1047 raft_consensus.cc:3058] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:25.874773 848 leader_election.cc:304] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.875267 1584 raft_consensus.cc:2747] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.860473 1171 ts_tablet_manager.cc:1428] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.001s sys 0.000s
I20250905 19:52:25.877735 1171 tablet_bootstrap.cc:492] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.877698 1047 raft_consensus.cc:2466] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:25.878466 849 leader_election.cc:304] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.880012 1584 raft_consensus.cc:2802] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:25.880507 1584 raft_consensus.cc:695] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:25.882869 1934 tablet_loader.cc:96] loaded metadata for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84])
I20250905 19:52:25.882598 1584 consensus_queue.cc:237] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.880098 2007 raft_consensus.cc:491] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.884586 2007 raft_consensus.cc:513] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.885136 1934 tablet_loader.cc:96] loaded metadata for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4])
I20250905 19:52:25.885171 1169 tablet_bootstrap.cc:492] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.885881 1171 tablet_bootstrap.cc:654] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.886686 1934 tablet_loader.cc:96] loaded metadata for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1 (table default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6])
I20250905 19:52:25.887181 1169 tablet_bootstrap.cc:654] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.888751 1798 raft_consensus.cc:491] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.888916 1798 raft_consensus.cc:513] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.889389 1934 tablet_loader.cc:96] loaded metadata for tablet e5fc2c2ba49444af939f5a975165362b (table default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b])
I20250905 19:52:25.889521 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ae83eea5f74e9187713e8b82c9a23b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:25.889623 1798 leader_election.cc:290] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:25.889842 1934 tablet_loader.cc:96] loaded metadata for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a [id=1598bab0cf8f41e0a4f7ee643cfeeaab])
W20250905 19:52:25.890188 848 leader_election.cc:343] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.890419 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:25.890807 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.890903 986 leader_election.cc:343] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.891278 984 leader_election.cc:343] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:25.891396 984 leader_election.cc:304] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.891777 1798 raft_consensus.cc:2747] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:25.891803 849 leader_election.cc:343] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b
I20250905 19:52:25.892009 849 leader_election.cc:304] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.892359 1934 tablet_loader.cc:96] loaded metadata for tablet e67ed7e9e5724de389504701584bd2b6 (table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923])
I20250905 19:52:25.892400 1672 raft_consensus.cc:2747] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.892792 2007 leader_election.cc:290] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.893278 1934 tablet_loader.cc:96] loaded metadata for tablet e6ac13b19fb6427a9ef35a5160033a54 (table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f])
I20250905 19:52:25.894263 1934 tablet_loader.cc:96] loaded metadata for tablet e7086c5bfa4644d08e4f55231b85acf4 (table default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39])
I20250905 19:52:25.899016 1584 raft_consensus.cc:491] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.899938 1584 raft_consensus.cc:513] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.901464 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d8144bb4f14358af539b7aae326918" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:25.902086 848 leader_election.cc:343] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:25.902225 849 leader_election.cc:343] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918
I20250905 19:52:25.902225 1934 tablet_loader.cc:96] loaded metadata for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151])
I20250905 19:52:25.902348 849 leader_election.cc:304] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:25.902643 1584 leader_election.cc:290] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:25.902911 2007 raft_consensus.cc:2747] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.903224 1934 tablet_loader.cc:96] loaded metadata for tablet edb9379335ca4eb5a32a6cbe85601a1f (table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4])
I20250905 19:52:25.905110 1170 tablet_bootstrap.cc:492] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.905355 1170 ts_tablet_manager.cc:1397] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.011s
I20250905 19:52:25.905596 1185 ts_tablet_manager.cc:1918] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:25.905740 1185 log.cc:1199] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/74a7e7d9874e4d26bda3ab24a6e24adc
I20250905 19:52:25.906139 1185 ts_tablet_manager.cc:1939] T 74a7e7d9874e4d26bda3ab24a6e24adc P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:25.906620 1170 raft_consensus.cc:357] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.906895 1170 raft_consensus.cc:383] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.906944 1185 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 458) took 1408 ms. Trace:
I20250905 19:52:25.907104 1170 raft_consensus.cc:738] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.907071 1185 rpcz_store.cc:276] 0905 19:52:24.498855 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:24.499712 (+ 857us) service_pool.cc:225] Handling call
0905 19:52:25.835139 (+1335427us) ts_tablet_manager.cc:1145] Deleting tablet 74a7e7d9874e4d26bda3ab24a6e24adc
0905 19:52:25.835155 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:25.889977 (+ 54822us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.905453 (+ 15476us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:25.906905 (+ 1452us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1335259}
W20250905 19:52:25.907388 847 connection.cc:441] server connection from 127.0.0.1:51212 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:51212 (request call id 458) could send its response
I20250905 19:52:25.907639 1170 consensus_queue.cc:260] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.908891 1171 tablet_bootstrap.cc:492] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.909162 1171 ts_tablet_manager.cc:1397] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.032s user 0.004s sys 0.000s
I20250905 19:52:25.910022 1171 raft_consensus.cc:357] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.910259 1171 raft_consensus.cc:383] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.910348 1171 raft_consensus.cc:738] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.910646 1171 consensus_queue.cc:260] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.910315 1170 ts_tablet_manager.cc:1428] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.000s sys 0.001s
I20250905 19:52:25.912465 1170 tablet_bootstrap.cc:492] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.915428 1171 ts_tablet_manager.cc:1428] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:25.918002 1170 tablet_bootstrap.cc:654] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.920485 1171 tablet_bootstrap.cc:492] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.922999 1171 tablet_bootstrap.cc:654] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.903946 1934 tablet_loader.cc:96] loaded metadata for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7])
I20250905 19:52:25.925272 1934 tablet_loader.cc:96] loaded metadata for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530])
I20250905 19:52:25.925467 1169 tablet_bootstrap.cc:492] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.925709 1169 ts_tablet_manager.cc:1397] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.041s user 0.007s sys 0.006s
I20250905 19:52:25.925776 1934 tablet_loader.cc:96] loaded metadata for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df])
I20250905 19:52:25.926414 1934 tablet_loader.cc:96] loaded metadata for tablet f2d929ec75d34cddad5286d8e37fcefb (table default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b])
I20250905 19:52:25.926805 1934 tablet_loader.cc:96] loaded metadata for tablet f34f0b29d6104cac8ab35309fd1edae8 (table default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09])
I20250905 19:52:25.926949 1169 raft_consensus.cc:357] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.927280 1169 raft_consensus.cc:383] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.927451 1169 raft_consensus.cc:738] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.927734 1934 tablet_loader.cc:96] loaded metadata for tablet f6f29ac5e68d4d8da8f3496fefe695ec (table default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3])
I20250905 19:52:25.927776 1169 consensus_queue.cc:260] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.928638 1934 tablet_loader.cc:96] loaded metadata for tablet fa5d34f9c0ee4d55ade61b30b0f4c293 (table default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9])
I20250905 19:52:25.929021 1934 tablet_loader.cc:96] loaded metadata for tablet fb72fe6049894527917e1481bcb7b03a (table default.Table_191f4f9b06184a8a8429cce5802c520e [id=a9604dfc1b44424f97da39857b13ae91])
I20250905 19:52:25.929617 1934 tablet_loader.cc:96] loaded metadata for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_6ec4f72266c7447da59b8544cfa98cd6 [id=0b6f3d59aebb4de0b34652b9f2c8ded7])
I20250905 19:52:25.929940 1934 tablet_loader.cc:96] loaded metadata for tablet fcb9bcc5e8634211a72cc05e936c71e7 (table default.Table_5bf03647f71d46a8b3d3f0b5bc385154 [id=e19dba4ecf6d4c22965720b89e7c45d4])
I20250905 19:52:25.932897 1934 tablet_loader.cc:96] loaded metadata for tablet ff2f289c293f41379e4af470d747860c (table default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5])
I20250905 19:52:25.933728 1934 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:52:25.938442 1169 ts_tablet_manager.cc:1428] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:25.943497 1169 tablet_bootstrap.cc:492] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.944969 1170 tablet_bootstrap.cc:492] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.945235 1170 ts_tablet_manager.cc:1397] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.003s
I20250905 19:52:25.945777 1169 tablet_bootstrap.cc:654] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.946274 1170 raft_consensus.cc:357] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.946557 1170 raft_consensus.cc:383] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.946681 1170 raft_consensus.cc:738] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.947003 1170 consensus_queue.cc:260] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.950624 1171 tablet_bootstrap.cc:492] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.950822 1171 ts_tablet_manager.cc:1397] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.030s user 0.002s sys 0.013s
I20250905 19:52:25.952641 1171 raft_consensus.cc:357] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.952833 1171 raft_consensus.cc:383] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.952913 1171 raft_consensus.cc:738] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.953188 1171 consensus_queue.cc:260] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.954340 1170 ts_tablet_manager.cc:1428] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250905 19:52:25.959427 1170 tablet_bootstrap.cc:492] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.961515 1170 tablet_bootstrap.cc:654] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.961791 1171 ts_tablet_manager.cc:1428] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:25.970549 1171 tablet_bootstrap.cc:492] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:25.971007 1169 tablet_bootstrap.cc:492] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:25.971276 1169 ts_tablet_manager.cc:1397] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.016s sys 0.002s
I20250905 19:52:25.972218 1169 raft_consensus.cc:357] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.972365 1171 tablet_bootstrap.cc:654] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.972544 1169 raft_consensus.cc:383] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.972687 1169 raft_consensus.cc:738] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.972994 1169 consensus_queue.cc:260] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.973640 1169 ts_tablet_manager.cc:1428] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:25.974747 1169 tablet_bootstrap.cc:492] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:25.976647 1169 tablet_bootstrap.cc:654] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:25.980199 1170 tablet_bootstrap.cc:492] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:25.980469 1170 ts_tablet_manager.cc:1397] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.000s
I20250905 19:52:25.981405 1170 raft_consensus.cc:357] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.981707 1170 raft_consensus.cc:383] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:25.981837 1170 raft_consensus.cc:738] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:25.982126 1170 consensus_queue.cc:260] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.982635 1934 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:25.982738 1934 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:52:25.986277 1170 ts_tablet_manager.cc:1428] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:25.990486 1849 raft_consensus.cc:491] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:25.990696 1849 raft_consensus.cc:513] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:25.991045 1170 tablet_bootstrap.cc:492] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:25.991328 1849 leader_election.cc:290] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:25.992872 712 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2f581403ec924fb0948400df41ed8aab
I20250905 19:52:25.993103 1170 tablet_bootstrap.cc:654] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:25.994817 711 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2f581403ec924fb0948400df41ed8aab
I20250905 19:52:25.994949 711 leader_election.cc:304] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:25.995379 1849 raft_consensus.cc:2747] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:25.998540 1171 tablet_bootstrap.cc:492] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:25.998767 1171 ts_tablet_manager.cc:1397] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.001s
I20250905 19:52:25.999826 1171 raft_consensus.cc:357] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.000147 1171 raft_consensus.cc:383] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.000249 1171 raft_consensus.cc:738] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.000568 1171 consensus_queue.cc:260] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.001408 1171 ts_tablet_manager.cc:1428] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250905 19:52:26.005415 1934 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:52:26.005894 1171 tablet_bootstrap.cc:492] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.011345 1171 tablet_bootstrap.cc:654] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.012236 1849 raft_consensus.cc:491] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.012486 1169 tablet_bootstrap.cc:492] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.012409 1849 raft_consensus.cc:513] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.012636 1169 ts_tablet_manager.cc:1397] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.003s
I20250905 19:52:26.013095 1849 leader_election.cc:290] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.013480 1169 raft_consensus.cc:357] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.013707 1169 raft_consensus.cc:383] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.013809 1169 raft_consensus.cc:738] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.014030 1169 consensus_queue.cc:260] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.014560 1169 ts_tablet_manager.cc:1428] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.014890 1169 tablet_bootstrap.cc:492] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:26.017323 711 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 01c20eea1b024d7f93f18f0c089efc42
W20250905 19:52:26.017678 712 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:26.017782 712 leader_election.cc:304] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.018433 1169 tablet_bootstrap.cc:654] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.020248 1849 raft_consensus.cc:2747] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.024686 1170 tablet_bootstrap.cc:492] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.024892 1170 ts_tablet_manager.cc:1397] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.037s user 0.008s sys 0.012s
I20250905 19:52:26.025707 1170 raft_consensus.cc:357] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.026042 1170 raft_consensus.cc:383] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.026145 1170 raft_consensus.cc:738] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.026430 1170 consensus_queue.cc:260] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.027112 1170 ts_tablet_manager.cc:1428] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:26.027732 1171 tablet_bootstrap.cc:492] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.027879 1171 ts_tablet_manager.cc:1397] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.007s sys 0.000s
I20250905 19:52:26.028043 1170 tablet_bootstrap.cc:492] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.028757 1171 raft_consensus.cc:357] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.028932 1171 raft_consensus.cc:383] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.029045 1171 raft_consensus.cc:738] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.029294 1171 consensus_queue.cc:260] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.030894 1169 tablet_bootstrap.cc:492] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.031057 1169 ts_tablet_manager.cc:1397] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.016s user 0.011s sys 0.000s
I20250905 19:52:26.031396 1171 ts_tablet_manager.cc:1428] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:26.032697 1169 raft_consensus.cc:357] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.032900 1169 raft_consensus.cc:383] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.032971 1169 raft_consensus.cc:738] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.034425 1171 tablet_bootstrap.cc:492] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.034512 1169 consensus_queue.cc:260] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.035022 1169 ts_tablet_manager.cc:1428] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:26.039489 1849 raft_consensus.cc:491] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.039592 1169 tablet_bootstrap.cc:492] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.039660 1849 raft_consensus.cc:513] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.040371 1849 leader_election.cc:290] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.042655 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.045825 1934 catalog_manager.cc:5966] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Loaded TSK: 0
I20250905 19:52:26.046172 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "544dffc8b77049c295921d0af091b17d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.047451 1169 tablet_bootstrap.cc:654] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.047457 1171 tablet_bootstrap.cc:654] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.048707 1170 tablet_bootstrap.cc:654] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.049644 1934 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:52:26.049925 1171 tablet_bootstrap.cc:492] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.050077 1171 ts_tablet_manager.cc:1397] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.002s
I20250905 19:52:26.050860 1171 raft_consensus.cc:357] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.051041 1171 raft_consensus.cc:383] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.051108 1171 raft_consensus.cc:738] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.051561 1171 consensus_queue.cc:260] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.053145 1171 ts_tablet_manager.cc:1428] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:26.056571 1171 tablet_bootstrap.cc:492] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.059425 1169 tablet_bootstrap.cc:492] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.059562 1169 ts_tablet_manager.cc:1397] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250905 19:52:26.060341 1169 raft_consensus.cc:357] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.060513 1169 raft_consensus.cc:383] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.060566 1169 raft_consensus.cc:738] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.060768 1169 consensus_queue.cc:260] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.061270 1169 ts_tablet_manager.cc:1428] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:26.061555 1169 tablet_bootstrap.cc:492] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.062736 1170 tablet_bootstrap.cc:492] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.062902 1170 ts_tablet_manager.cc:1397] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250905 19:52:26.063707 1170 raft_consensus.cc:357] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.063898 1170 raft_consensus.cc:383] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.064013 1170 raft_consensus.cc:738] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.064275 1170 consensus_queue.cc:260] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.067749 1170 ts_tablet_manager.cc:1428] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250905 19:52:26.071781 1171 tablet_bootstrap.cc:654] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.077759 711 leader_election.cc:304] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.078112 1849 raft_consensus.cc:2747] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.078879 1170 tablet_bootstrap.cc:492] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.080699 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:26.089515 1170 tablet_bootstrap.cc:654] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.090193 1540 master_service.cc:948] ReplaceTablet: received request to replace tablet 3c824f5591044c4198a51a25661f71e5 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:26.094108 1169 tablet_bootstrap.cc:654] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.089751 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:26.095870 2035 tablet_replica.cc:331] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:26.096117 1542 master_service.cc:948] ReplaceTablet: received request to replace tablet 6f3473785fff45d692bd50b1015ffe39 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:26.096155 2035 raft_consensus.cc:2241] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:26.096813 1546 master_service.cc:948] ReplaceTablet: received request to replace tablet 03a12cf719f34096bb82f5846bbe3b2d from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:26.096854 2035 raft_consensus.cc:2270] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:26.097522 1543 master_service.cc:948] ReplaceTablet: received request to replace tablet 9630f1feb6d74472b5f6f4b54151311c from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:26.098095 1547 master_service.cc:948] ReplaceTablet: received request to replace tablet 421b28d047124222ae9cbc603ec22cee from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:26.092468 1539 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_5277901ee519486abb3c0e90d83961b5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:26.092840 1536 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:26.093119 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_487dad5063df425c8b4a44d7f8adbd39"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:26.093446 1522 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_cfe02f65c42d492b9ebd6d9891ff4b34"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:26.093760 1517 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_6e9009b9124c413eb5058a1f3294990a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:26.105295 1539 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5277901ee519486abb3c0e90d83961b5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:26.105690 1517 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e9009b9124c413eb5058a1f3294990a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:26.106556 1522 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_cfe02f65c42d492b9ebd6d9891ff4b34 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:26.107790 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_487dad5063df425c8b4a44d7f8adbd39 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:26.109079 1536 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:26.110783 2035 ts_tablet_manager.cc:1905] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:26.114326 1543 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:26.115537 1543 catalog_manager.cc:6652] ReplaceTablet: tablet 9630f1feb6d74472b5f6f4b54151311c deleted and replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e
I20250905 19:52:26.117033 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:26.118105 1546 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:26.118577 1546 catalog_manager.cc:6652] ReplaceTablet: tablet 03a12cf719f34096bb82f5846bbe3b2d deleted and replaced by tablet 03cf686811e041e7a001669b3d86e28a
I20250905 19:52:26.118961 1798 raft_consensus.cc:491] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.119115 1798 raft_consensus.cc:513] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.119325 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:26.119776 1798 leader_election.cc:290] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.120695 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3805127dd78489e8545da7414ead904" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.121326 984 leader_election.cc:343] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.142823 770 raft_consensus.cc:3058] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.148559 1849 raft_consensus.cc:491] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.148777 1849 raft_consensus.cc:513] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.151201 1849 leader_election.cc:290] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:26.151422 711 leader_election.cc:343] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 74a7e7d9874e4d26bda3ab24a6e24adc
I20250905 19:52:26.154189 2035 ts_tablet_manager.cc:1918] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:26.161938 2035 log.cc:1199] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/262bcd1cfc994bb58e999b141b9bc7ea
I20250905 19:52:26.162458 2035 ts_tablet_manager.cc:1939] T 262bcd1cfc994bb58e999b141b9bc7ea P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:26.157780 1170 tablet_bootstrap.cc:492] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.163623 1672 raft_consensus.cc:491] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.163846 1672 raft_consensus.cc:513] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.164801 1672 leader_election.cc:290] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.168890 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3805127dd78489e8545da7414ead904" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.169656 986 leader_election.cc:343] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.166940 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "415508e72ee34bc391c4369b5f8a2f48" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.169862 986 leader_election.cc:304] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.163544 1170 ts_tablet_manager.cc:1397] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.096s user 0.007s sys 0.003s
I20250905 19:52:26.170464 1798 raft_consensus.cc:2747] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.171392 1170 raft_consensus.cc:357] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.171684 1170 raft_consensus.cc:383] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.171782 1170 raft_consensus.cc:738] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.172120 1170 consensus_queue.cc:260] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.172906 1169 tablet_bootstrap.cc:492] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.173079 1169 ts_tablet_manager.cc:1397] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.112s user 0.007s sys 0.002s
I20250905 19:52:26.174005 1169 raft_consensus.cc:357] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.174245 1169 raft_consensus.cc:383] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.174312 1169 raft_consensus.cc:738] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.174567 1169 consensus_queue.cc:260] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.175216 1169 ts_tablet_manager.cc:1428] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.181311 1584 raft_consensus.cc:491] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.181648 1584 raft_consensus.cc:513] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.182293 1584 leader_election.cc:290] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.183627 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "415508e72ee34bc391c4369b5f8a2f48" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.183807 1169 tablet_bootstrap.cc:492] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.183909 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff2f289c293f41379e4af470d747860c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.184080 1050 raft_consensus.cc:2466] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.184345 1849 raft_consensus.cc:491] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.184523 1849 raft_consensus.cc:513] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.185122 1849 leader_election.cc:290] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.185374 1169 tablet_bootstrap.cc:654] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.183830 1047 raft_consensus.cc:2466] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:26.186483 712 leader_election.cc:343] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 74a7e7d9874e4d26bda3ab24a6e24adc
I20250905 19:52:26.186614 712 leader_election.cc:304] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.187515 1991 raft_consensus.cc:2747] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.187755 1991 raft_consensus.cc:491] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.187887 1991 raft_consensus.cc:513] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.188609 1991 leader_election.cc:290] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.188870 849 leader_election.cc:304] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:26.189133 712 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:26.189596 1658 raft_consensus.cc:2802] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.189707 1658 raft_consensus.cc:491] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.189812 1658 raft_consensus.cc:3058] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.191498 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:26.192342 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:26.193027 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:26.193501 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 9630f1feb6d74472b5f6f4b54151311c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9630f1feb6d74472b5f6f4b54151311c already in progress: creating tablet
W20250905 19:52:26.193940 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 03a12cf719f34096bb82f5846bbe3b2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03a12cf719f34096bb82f5846bbe3b2d already in progress: creating tablet
W20250905 19:52:26.194188 711 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:26.194386 711 leader_election.cc:304] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.194967 1849 raft_consensus.cc:2747] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.195315 1798 raft_consensus.cc:491] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.195477 1798 raft_consensus.cc:513] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.196295 1798 leader_election.cc:290] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.196929 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac2e51a9071c4e1dae9e147687291175" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.197204 913 raft_consensus.cc:2466] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:26.197751 986 leader_election.cc:304] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:26.198578 1658 raft_consensus.cc:513] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.198969 1798 raft_consensus.cc:2802] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.199079 1798 raft_consensus.cc:491] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.199159 1798 raft_consensus.cc:3058] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.200966 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:26.201259 1184 tablet_replica.cc:331] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:26.201509 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:26.201529 1184 raft_consensus.cc:2241] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:26.204211 1184 raft_consensus.cc:2270] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:26.204928 1184 ts_tablet_manager.cc:1905] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:26.214340 1184 ts_tablet_manager.cc:1918] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:26.214520 1184 log.cc:1199] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:26.214972 1184 ts_tablet_manager.cc:1939] T 9630f1feb6d74472b5f6f4b54151311c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:26.216822 1658 leader_election.cc:290] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.203783 1798 raft_consensus.cc:513] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.218604 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac2e51a9071c4e1dae9e147687291175" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.219367 984 leader_election.cc:343] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.224444 1546 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_c1b824c37b3b4413b20c5b2172059490"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:26.226817 1546 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c1b824c37b3b4413b20c5b2172059490 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:26.228470 1543 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_da99ffef6e2c4d60a0fa49e3fb2ad61a" } new_table_name: "default.Table_fd318ab664334e4bb9d57d696f6449d2" modify_external_catalogs: true
I20250905 19:52:26.230986 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac2e51a9071c4e1dae9e147687291175" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.231851 984 leader_election.cc:343] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.233186 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac2e51a9071c4e1dae9e147687291175" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:26.233480 913 raft_consensus.cc:3058] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.237435 913 raft_consensus.cc:2466] T ac2e51a9071c4e1dae9e147687291175 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:26.238684 986 leader_election.cc:304] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.239097 1997 raft_consensus.cc:2802] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.239312 1997 raft_consensus.cc:695] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:26.239378 1798 leader_election.cc:290] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.239673 1997 consensus_queue.cc:237] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.240947 1849 raft_consensus.cc:491] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.241183 1849 raft_consensus.cc:513] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.242003 1849 leader_election.cc:290] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.243152 1999 raft_consensus.cc:491] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.243327 1999 raft_consensus.cc:513] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.244120 1999 leader_election.cc:290] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:26.244571 848 leader_election.cc:343] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.245585 711 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
W20250905 19:52:26.246529 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 72040.8us
I20250905 19:52:26.247699 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff2f289c293f41379e4af470d747860c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.248037 848 consensus_peers.cc:489] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:26.248373 848 leader_election.cc:343] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.248387 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff2f289c293f41379e4af470d747860c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.248786 1584 raft_consensus.cc:491] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.248919 1584 raft_consensus.cc:513] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.249577 1584 leader_election.cc:290] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.250591 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.250977 848 leader_election.cc:343] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.251441 848 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.251742 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723168b075874dfcb3eb50c5134c499e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.252245 986 leader_election.cc:343] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.256117 849 leader_election.cc:304] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:26.256464 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 67285.7us
I20250905 19:52:26.256557 1658 raft_consensus.cc:2802] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.256747 1658 raft_consensus.cc:491] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.256906 1658 raft_consensus.cc:3058] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.258369 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ff2f289c293f41379e4af470d747860c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.258525 1050 raft_consensus.cc:3058] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.259683 1658 raft_consensus.cc:513] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.260159 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723168b075874dfcb3eb50c5134c499e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.260742 984 leader_election.cc:343] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.260908 984 leader_election.cc:304] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.261129 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "415508e72ee34bc391c4369b5f8a2f48" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:26.261243 1050 raft_consensus.cc:2466] T ff2f289c293f41379e4af470d747860c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.261420 1997 raft_consensus.cc:2747] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:26.261564 848 leader_election.cc:343] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.261816 849 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:26.261924 849 leader_election.cc:304] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.262019 1658 leader_election.cc:290] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.262180 1658 raft_consensus.cc:2747] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.262414 849 leader_election.cc:304] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.262643 1672 consensus_queue.cc:1035] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:26.262954 712 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
I20250905 19:52:26.263060 1672 raft_consensus.cc:2802] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.263100 712 leader_election.cc:304] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.263187 1672 raft_consensus.cc:695] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.263105 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "415508e72ee34bc391c4369b5f8a2f48" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.263309 1050 raft_consensus.cc:3058] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.263448 1672 consensus_queue.cc:237] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.263895 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f732ae0627c4da5b1691cde9b4d5562" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.265292 1849 raft_consensus.cc:2747] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.266101 712 leader_election.cc:304] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.266584 1050 raft_consensus.cc:2466] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.266611 1170 ts_tablet_manager.cc:1428] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.096s user 0.002s sys 0.000s
I20250905 19:52:26.266940 1170 tablet_bootstrap.cc:492] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.267098 849 leader_election.cc:304] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.267482 1973 raft_consensus.cc:2802] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.267619 1973 raft_consensus.cc:695] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.267900 1973 consensus_queue.cc:237] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.268191 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 262bcd1cfc994bb58e999b141b9bc7ea (table default.Table_7df10c45b04b4316ad24e8e2d141de9b [id=8f60da23a0864f92ba7c7ffcfa65277f]) successfully deleted
I20250905 19:52:26.268764 1170 tablet_bootstrap.cc:654] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.270910 1171 tablet_bootstrap.cc:492] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.271065 1171 ts_tablet_manager.cc:1397] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.218s user 0.051s sys 0.090s
I20250905 19:52:26.272022 1171 raft_consensus.cc:357] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.272226 1171 raft_consensus.cc:383] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.272318 1171 raft_consensus.cc:738] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.272436 1849 raft_consensus.cc:2747] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.272516 1171 consensus_queue.cc:260] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.281770 1171 ts_tablet_manager.cc:1428] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.000s sys 0.002s
I20250905 19:52:26.282091 1171 tablet_bootstrap.cc:492] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:26.282904 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9630f1feb6d74472b5f6f4b54151311c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9630f1feb6d74472b5f6f4b54151311c already in progress: creating tablet
W20250905 19:52:26.283257 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 03a12cf719f34096bb82f5846bbe3b2d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 03a12cf719f34096bb82f5846bbe3b2d already in progress: creating tablet
I20250905 19:52:26.284305 1171 tablet_bootstrap.cc:654] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.287535 1547 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 421b28d047124222ae9cbc603ec22cee
I20250905 19:52:26.288177 1547 catalog_manager.cc:6652] ReplaceTablet: tablet 421b28d047124222ae9cbc603ec22cee deleted and replaced by tablet 5a5d38136c034b8497045176f749a98d
I20250905 19:52:26.290751 1997 raft_consensus.cc:491] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.290899 1997 raft_consensus.cc:513] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.291522 1997 leader_election.cc:290] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.292263 1540 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 3c824f5591044c4198a51a25661f71e5
I20250905 19:52:26.292660 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.292801 1540 catalog_manager.cc:6652] ReplaceTablet: tablet 3c824f5591044c4198a51a25661f71e5 deleted and replaced by tablet 00d1d24868204ce09a7f5e40357fec18
W20250905 19:52:26.293251 986 leader_election.cc:343] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.295825 1171 tablet_bootstrap.cc:492] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.296037 1171 ts_tablet_manager.cc:1397] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.001s
I20250905 19:52:26.296787 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:26.296792 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:26.296988 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:26.297041 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:26.297916 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421b28d047124222ae9cbc603ec22cee already in progress: creating tablet
I20250905 19:52:26.298080 1171 raft_consensus.cc:357] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:26.298199 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
I20250905 19:52:26.298333 1171 raft_consensus.cc:383] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.298413 1171 raft_consensus.cc:738] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.298707 1171 consensus_queue.cc:260] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.299434 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:26.299721 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:26.300590 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421b28d047124222ae9cbc603ec22cee already in progress: creating tablet
I20250905 19:52:26.300602 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.300859 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
W20250905 19:52:26.301074 984 leader_election.cc:343] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.301209 1171 ts_tablet_manager.cc:1428] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:26.301206 984 leader_election.cc:304] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.301522 1171 tablet_bootstrap.cc:492] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.301792 1997 raft_consensus.cc:2747] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.303494 1171 tablet_bootstrap.cc:654] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.303829 1170 tablet_bootstrap.cc:492] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.304113 1170 ts_tablet_manager.cc:1397] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250905 19:52:26.305130 1170 raft_consensus.cc:357] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.305373 1170 raft_consensus.cc:383] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.305486 1170 raft_consensus.cc:738] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.305804 1170 consensus_queue.cc:260] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.308377 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:26.309660 1522 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_4447fb37960e42dfa2c6eb8ced9c6b68" } new_table_name: "default.Table_a82a3756e48a4bfc99f4aacf9b57f6d3" modify_external_catalogs: true
I20250905 19:52:26.310516 1517 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_af469c2f14f24b01ae62bc25f1e8e4f8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:26.311093 1517 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_af469c2f14f24b01ae62bc25f1e8e4f8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:26.312084 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:26.312198 1540 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_849b4e364b3f4ff2b47a2347f84d6b0d" } new_table_name: "default.Table_90ee22819c074028aaf36f24d7de572d" modify_external_catalogs: true
I20250905 19:52:26.313234 1171 tablet_bootstrap.cc:492] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.313405 1171 ts_tablet_manager.cc:1397] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20250905 19:52:26.313349 1539 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_9da2eedb50e64356ba6a49d93bafbe28" } new_table_name: "default.Table_7afe807a480844efac1cb503fbba94ae" modify_external_catalogs: true
I20250905 19:52:26.314337 1171 raft_consensus.cc:357] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.314530 1171 raft_consensus.cc:383] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.314630 1171 raft_consensus.cc:738] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.315106 1171 consensus_queue.cc:260] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.316001 1171 ts_tablet_manager.cc:1428] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:26.316361 1171 tablet_bootstrap.cc:492] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.318810 1171 tablet_bootstrap.cc:654] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.324383 1184 tablet_replica.cc:331] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:26.324604 1184 raft_consensus.cc:2241] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:26.324770 1184 raft_consensus.cc:2270] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:26.325845 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9630f1feb6d74472b5f6f4b54151311c (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]) successfully deleted
I20250905 19:52:26.333798 1542 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:26.334481 1542 catalog_manager.cc:6652] ReplaceTablet: tablet 6f3473785fff45d692bd50b1015ffe39 deleted and replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6
I20250905 19:52:26.339985 1997 raft_consensus.cc:491] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.340199 1997 raft_consensus.cc:513] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.340922 1997 leader_election.cc:290] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.341174 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:26.342607 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44769f1170354870ae2d34fdb67ffbae" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.343674 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:26.345544 986 leader_election.cc:343] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.345973 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6f3473785fff45d692bd50b1015ffe39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f3473785fff45d692bd50b1015ffe39 already in progress: creating tablet
I20250905 19:52:26.347584 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:26.348575 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6f3473785fff45d692bd50b1015ffe39 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6f3473785fff45d692bd50b1015ffe39 already in progress: creating tablet
I20250905 19:52:26.359326 1542 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_37507bfcf78648779f20bfe755073f99"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:26.360030 1542 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_37507bfcf78648779f20bfe755073f99 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:26.361508 1536 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_55ca8eccc0224578835d4f9f8fae2311" } modify_external_catalogs: true
I20250905 19:52:26.361716 1536 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_55ca8eccc0224578835d4f9f8fae2311" } modify_external_catalogs: true
I20250905 19:52:26.364441 1170 ts_tablet_manager.cc:1428] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.060s user 0.002s sys 0.000s
I20250905 19:52:26.365057 1184 ts_tablet_manager.cc:1905] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:26.366492 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44769f1170354870ae2d34fdb67ffbae" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.367861 984 leader_election.cc:343] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.368041 984 leader_election.cc:304] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.368537 1997 raft_consensus.cc:2747] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.370252 1170 tablet_bootstrap.cc:492] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.377813 1170 tablet_bootstrap.cc:654] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.351267 1171 tablet_bootstrap.cc:492] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.379814 1171 ts_tablet_manager.cc:1397] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.064s user 0.000s sys 0.004s
I20250905 19:52:26.380702 1171 raft_consensus.cc:357] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.380935 1171 raft_consensus.cc:383] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.381028 1171 raft_consensus.cc:738] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.381325 1171 consensus_queue.cc:260] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.381757 1169 tablet_bootstrap.cc:492] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.381853 1171 ts_tablet_manager.cc:1428] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:26.381889 1169 ts_tablet_manager.cc:1397] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.198s user 0.032s sys 0.085s
I20250905 19:52:26.382498 1169 raft_consensus.cc:357] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.382671 1169 raft_consensus.cc:383] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.382746 1169 raft_consensus.cc:738] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.384341 1169 consensus_queue.cc:260] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.385538 1169 ts_tablet_manager.cc:1428] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:26.386835 1169 tablet_bootstrap.cc:492] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.389533 1169 tablet_bootstrap.cc:654] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.390650 1184 ts_tablet_manager.cc:1918] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:26.390813 1184 log.cc:1199] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:26.391211 1184 ts_tablet_manager.cc:1939] T 03a12cf719f34096bb82f5846bbe3b2d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:26.392815 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 03a12cf719f34096bb82f5846bbe3b2d (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]) successfully deleted
I20250905 19:52:26.395162 1171 tablet_bootstrap.cc:492] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.396445 1169 tablet_bootstrap.cc:492] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.396585 1169 ts_tablet_manager.cc:1397] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.000s
I20250905 19:52:26.397282 1169 raft_consensus.cc:357] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.397468 1169 raft_consensus.cc:383] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.397523 1169 raft_consensus.cc:738] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.399570 1169 consensus_queue.cc:260] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.400697 1169 ts_tablet_manager.cc:1428] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250905 19:52:26.401119 1171 tablet_bootstrap.cc:654] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.400974 1169 tablet_bootstrap.cc:492] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.402734 1973 consensus_queue.cc:1035] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:26.403586 1169 tablet_bootstrap.cc:654] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.409240 1184 tablet_replica.cc:331] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:26.410255 1170 tablet_bootstrap.cc:492] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.410262 1171 tablet_bootstrap.cc:492] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.410419 1171 ts_tablet_manager.cc:1397] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20250905 19:52:26.410560 1170 ts_tablet_manager.cc:1397] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.040s user 0.005s sys 0.001s
I20250905 19:52:26.411310 1171 raft_consensus.cc:357] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.411335 1170 raft_consensus.cc:357] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.411518 1171 raft_consensus.cc:383] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.411619 1171 raft_consensus.cc:738] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.411662 1170 raft_consensus.cc:383] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.411836 1170 raft_consensus.cc:738] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.412413 1584 raft_consensus.cc:491] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.412623 1584 raft_consensus.cc:513] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.413443 1584 leader_election.cc:290] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.413951 1171 consensus_queue.cc:260] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.415057 1171 ts_tablet_manager.cc:1428] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250905 19:52:26.415364 1171 tablet_bootstrap.cc:492] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.416029 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.417107 1171 tablet_bootstrap.cc:654] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:26.419615 848 leader_election.cc:343] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.419868 848 consensus_peers.cc:489] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:26.421525 1170 consensus_queue.cc:260] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.422458 1170 ts_tablet_manager.cc:1428] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
W20250905 19:52:26.422899 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421b28d047124222ae9cbc603ec22cee already in progress: creating tablet
I20250905 19:52:26.423102 1170 tablet_bootstrap.cc:492] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:26.423336 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
I20250905 19:52:26.409284 1169 tablet_bootstrap.cc:492] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.423715 1169 ts_tablet_manager.cc:1397] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.001s
I20250905 19:52:26.424520 1169 raft_consensus.cc:357] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.424703 1169 raft_consensus.cc:383] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.424767 1169 raft_consensus.cc:738] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.425000 1169 consensus_queue.cc:260] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.425504 1169 ts_tablet_manager.cc:1428] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:26.425807 1169 tablet_bootstrap.cc:492] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.427500 1169 tablet_bootstrap.cc:654] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.417773 1999 raft_consensus.cc:491] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.420843 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.428130 1047 raft_consensus.cc:2466] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.428078 1999 raft_consensus.cc:513] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.428618 849 leader_election.cc:304] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.428680 1999 leader_election.cc:290] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.429080 1849 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.429226 1849 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.429823 1849 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.429869 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.430456 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.430917 712 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
W20250905 19:52:26.431249 986 leader_election.cc:343] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.431493 984 leader_election.cc:343] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.431596 984 leader_election.cc:304] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.432189 1658 raft_consensus.cc:2802] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.432338 1658 raft_consensus.cc:491] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.432427 1658 raft_consensus.cc:3058] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.423719 1184 raft_consensus.cc:2241] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:26.433156 1184 raft_consensus.cc:2270] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:26.433662 1997 raft_consensus.cc:2747] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.433750 1184 ts_tablet_manager.cc:1905] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:26.435334 1658 raft_consensus.cc:513] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.436395 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.436563 1050 raft_consensus.cc:3058] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.438072 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.438683 848 leader_election.cc:343] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.439081 1050 raft_consensus.cc:2466] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.439775 849 leader_election.cc:304] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.440227 1672 raft_consensus.cc:2802] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.440889 1672 raft_consensus.cc:695] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.441471 1672 consensus_queue.cc:237] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:26.445216 711 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:26.445364 711 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.445560 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38687
--webserver_interface=127.0.104.62
--webserver_port=46537
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 19:52:26.446102 1849 raft_consensus.cc:2747] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.447342 1171 tablet_bootstrap.cc:492] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.447512 1171 ts_tablet_manager.cc:1397] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.032s user 0.000s sys 0.003s
I20250905 19:52:26.425616 1170 tablet_bootstrap.cc:654] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.448352 1171 raft_consensus.cc:357] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.448561 1171 raft_consensus.cc:383] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.448642 1171 raft_consensus.cc:738] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.448901 1171 consensus_queue.cc:260] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.449394 1171 ts_tablet_manager.cc:1428] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:26.449720 1171 tablet_bootstrap.cc:492] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.451429 1171 tablet_bootstrap.cc:654] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.441960 1658 leader_election.cc:290] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.454075 1658 raft_consensus.cc:491] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.454550 1658 raft_consensus.cc:513] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.462827 1546 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_f93e2f3cbd884fbf84082156519631cb" } modify_external_catalogs: true
I20250905 19:52:26.463152 1546 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_f93e2f3cbd884fbf84082156519631cb" } modify_external_catalogs: true
I20250905 19:52:26.463598 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.463697 1184 ts_tablet_manager.cc:1918] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:26.463806 1047 raft_consensus.cc:2466] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.463837 1542 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_7db3ab9bdfad4b35aa7c826807fc0b0a" } new_table_name: "default.Table_fe9af09b80d04739a6df4abf75c2d717" modify_external_catalogs: true
I20250905 19:52:26.464376 849 leader_election.cc:304] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:26.465260 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.465863 848 leader_election.cc:343] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.466048 1973 raft_consensus.cc:2802] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.466157 1973 raft_consensus.cc:491] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.466212 1658 leader_election.cc:290] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.466383 1973 raft_consensus.cc:3058] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.472160 1973 raft_consensus.cc:513] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.473357 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.474370 848 leader_election.cc:343] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.475270 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.475461 1047 raft_consensus.cc:3058] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.475790 1170 tablet_bootstrap.cc:492] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.475924 1170 ts_tablet_manager.cc:1397] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20250905 19:52:26.476755 1170 raft_consensus.cc:357] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.476918 1170 raft_consensus.cc:383] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.476995 1170 raft_consensus.cc:738] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.477252 1170 consensus_queue.cc:260] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.478068 1047 raft_consensus.cc:2466] T 4a518797d26140f5b9abe41eab415040 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.478536 849 leader_election.cc:304] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.478861 1672 raft_consensus.cc:2802] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.479386 1672 raft_consensus.cc:695] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.479794 1973 leader_election.cc:290] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.480050 1672 consensus_queue.cc:237] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:26.484047 848 consensus_peers.cc:489] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:26.484573 1170 ts_tablet_manager.cc:1428] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250905 19:52:26.488205 749 tablet_service.cc:1468] Processing CreateTablet for tablet e67ed7e9e5724de389504701584bd2b6 (DEFAULT_TABLE table=default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.488785 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67ed7e9e5724de389504701584bd2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.489146 1170 tablet_bootstrap.cc:492] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.490921 1170 tablet_bootstrap.cc:654] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.491626 1997 raft_consensus.cc:491] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.463814 1184 log.cc:1199] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:26.491811 1997 raft_consensus.cc:513] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.492157 1184 ts_tablet_manager.cc:1939] T 6f3473785fff45d692bd50b1015ffe39 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:26.492638 1997 leader_election.cc:290] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.493041 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfd03ad0bd34b6ca86e7b07a62b2a8e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.493517 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfd03ad0bd34b6ca86e7b07a62b2a8e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.493587 984 leader_election.cc:343] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.494074 986 leader_election.cc:343] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.494171 986 leader_election.cc:304] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.495569 1997 raft_consensus.cc:2747] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.498780 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6f3473785fff45d692bd50b1015ffe39 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]) successfully deleted
I20250905 19:52:26.500994 1171 tablet_bootstrap.cc:492] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.501147 1171 ts_tablet_manager.cc:1397] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.052s user 0.000s sys 0.004s
I20250905 19:52:26.501856 1849 raft_consensus.cc:491] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.502018 1849 raft_consensus.cc:513] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.502175 1171 raft_consensus.cc:357] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.502343 1171 raft_consensus.cc:383] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.502425 1171 raft_consensus.cc:738] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.502671 1171 consensus_queue.cc:260] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.502765 1849 leader_election.cc:290] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:26.504534 711 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 68f2ccd8a7a64b89b44f2d6767ec3466
W20250905 19:52:26.504773 712 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:26.504899 712 leader_election.cc:304] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.505288 1849 raft_consensus.cc:2747] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.505748 1170 tablet_bootstrap.cc:492] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.505894 1170 ts_tablet_manager.cc:1397] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20250905 19:52:26.506716 1170 raft_consensus.cc:357] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.506911 1170 raft_consensus.cc:383] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.506996 1170 raft_consensus.cc:738] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.507159 1171 ts_tablet_manager.cc:1428] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20250905 19:52:26.507284 1170 consensus_queue.cc:260] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.507597 1171 tablet_bootstrap.cc:492] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.509356 1171 tablet_bootstrap.cc:654] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.513572 1170 ts_tablet_manager.cc:1428] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 19:52:26.513689 891 tablet_service.cc:1468] Processing CreateTablet for tablet e67ed7e9e5724de389504701584bd2b6 (DEFAULT_TABLE table=default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.514117 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67ed7e9e5724de389504701584bd2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.516032 1169 tablet_bootstrap.cc:492] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.516178 1169 ts_tablet_manager.cc:1397] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.090s user 0.017s sys 0.026s
I20250905 19:52:26.517874 1849 raft_consensus.cc:491] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.518030 1849 raft_consensus.cc:513] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.518342 1169 raft_consensus.cc:357] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.518510 1169 raft_consensus.cc:383] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.518568 1169 raft_consensus.cc:738] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.518711 1849 leader_election.cc:290] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.518780 1169 consensus_queue.cc:260] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.519387 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.519918 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9a81376d4c0b47a9817deae4d34ffb47" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.519989 1169 ts_tablet_manager.cc:1428] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:26.520380 1169 tablet_bootstrap.cc:492] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.520684 711 leader_election.cc:304] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.521708 1849 raft_consensus.cc:2747] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.522564 1658 raft_consensus.cc:491] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.522642 1169 tablet_bootstrap.cc:654] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.522821 1658 raft_consensus.cc:513] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.533138 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.533231 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.533492 1047 raft_consensus.cc:2466] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:26.533716 848 leader_election.cc:343] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.534168 849 leader_election.cc:304] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.534825 1973 raft_consensus.cc:2802] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.534965 1973 raft_consensus.cc:491] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.535068 1973 raft_consensus.cc:3058] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.537662 1170 tablet_bootstrap.cc:492] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.538322 1171 tablet_bootstrap.cc:492] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.538511 1171 ts_tablet_manager.cc:1397] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.000s
I20250905 19:52:26.539350 1171 raft_consensus.cc:357] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.539476 1170 tablet_bootstrap.cc:654] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.539573 1171 raft_consensus.cc:383] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.539660 1171 raft_consensus.cc:738] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.539453 1973 raft_consensus.cc:513] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.539908 1171 consensus_queue.cc:260] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.540473 1171 ts_tablet_manager.cc:1428] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:26.540768 1171 tablet_bootstrap.cc:492] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.542508 1171 tablet_bootstrap.cc:654] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.543839 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.544664 848 leader_election.cc:343] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.545612 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.545779 1047 raft_consensus.cc:3058] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.548089 1973 leader_election.cc:290] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.548908 1047 raft_consensus.cc:2466] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.549551 849 leader_election.cc:304] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.550079 1658 leader_election.cc:290] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.550493 1658 raft_consensus.cc:2802] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.550793 1658 raft_consensus.cc:695] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
W20250905 19:52:26.551822 984 consensus_peers.cc:489] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:26.553351 1658 consensus_queue.cc:237] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.557618 1169 tablet_bootstrap.cc:492] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.557785 1169 ts_tablet_manager.cc:1397] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.037s user 0.021s sys 0.011s
I20250905 19:52:26.558720 1169 raft_consensus.cc:357] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.558933 1169 raft_consensus.cc:383] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.558983 1169 raft_consensus.cc:738] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.559198 1169 consensus_queue.cc:260] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.559923 1169 ts_tablet_manager.cc:1428] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:26.560380 1169 tablet_bootstrap.cc:492] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.561204 1171 tablet_bootstrap.cc:492] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.561362 1171 ts_tablet_manager.cc:1397] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.021s user 0.002s sys 0.001s
I20250905 19:52:26.563165 1973 raft_consensus.cc:491] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.563465 1169 tablet_bootstrap.cc:654] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.563370 1973 raft_consensus.cc:513] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.563925 1170 tablet_bootstrap.cc:492] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.564548 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.564678 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.564898 1047 raft_consensus.cc:2466] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.565588 849 leader_election.cc:304] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:26.565613 1171 raft_consensus.cc:357] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.565814 1171 raft_consensus.cc:383] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.565902 1171 raft_consensus.cc:738] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.565980 1584 raft_consensus.cc:2802] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.566089 1584 raft_consensus.cc:491] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.566134 1171 consensus_queue.cc:260] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.566222 1584 raft_consensus.cc:3058] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.566823 1171 ts_tablet_manager.cc:1428] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:26.568184 1849 raft_consensus.cc:491] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.568338 1849 raft_consensus.cc:513] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.569029 1849 leader_election.cc:290] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:26.569451 848 leader_election.cc:343] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.569890 1171 tablet_bootstrap.cc:492] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.570629 1584 raft_consensus.cc:513] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.570845 1170 ts_tablet_manager.cc:1397] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.002s
W20250905 19:52:26.571251 712 leader_election.cc:343] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:26.571748 1584 leader_election.cc:290] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.572098 1171 tablet_bootstrap.cc:654] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.572052 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.572613 848 leader_election.cc:343] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.572597 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.572817 1050 raft_consensus.cc:3058] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.573405 1170 raft_consensus.cc:357] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.573566 1170 raft_consensus.cc:383] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.573637 1170 raft_consensus.cc:738] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.573853 1170 consensus_queue.cc:260] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.576864 1050 raft_consensus.cc:2466] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.577329 1170 ts_tablet_manager.cc:1428] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250905 19:52:26.577529 849 leader_election.cc:304] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.578510 1170 tablet_bootstrap.cc:492] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:26.578645 711 leader_election.cc:343] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:26.578814 711 leader_election.cc:304] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.579407 1849 raft_consensus.cc:2747] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.579881 1584 raft_consensus.cc:2802] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.580178 1584 raft_consensus.cc:695] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.582749 1170 tablet_bootstrap.cc:654] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.580631 1584 consensus_queue.cc:237] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.584370 1973 leader_election.cc:290] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.594058 1171 tablet_bootstrap.cc:492] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.594275 1171 ts_tablet_manager.cc:1397] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250905 19:52:26.595242 1171 raft_consensus.cc:357] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.595541 1171 raft_consensus.cc:383] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.595636 1171 raft_consensus.cc:738] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.596009 1171 consensus_queue.cc:260] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.596623 1171 ts_tablet_manager.cc:1428] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250905 19:52:26.596973 1171 tablet_bootstrap.cc:492] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.598791 1171 tablet_bootstrap.cc:654] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.599289 1022 tablet_service.cc:1468] Processing CreateTablet for tablet e67ed7e9e5724de389504701584bd2b6 (DEFAULT_TABLE table=default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.599713 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e67ed7e9e5724de389504701584bd2b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.599511 1170 tablet_bootstrap.cc:492] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.599926 1170 ts_tablet_manager.cc:1397] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.003s
I20250905 19:52:26.600432 891 tablet_service.cc:1468] Processing CreateTablet for tablet 03cf686811e041e7a001669b3d86e28a (DEFAULT_TABLE table=default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.600775 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cf686811e041e7a001669b3d86e28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.600679 1170 raft_consensus.cc:357] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.600872 1170 raft_consensus.cc:383] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.600935 1170 raft_consensus.cc:738] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.601193 1170 consensus_queue.cc:260] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.602654 1170 ts_tablet_manager.cc:1428] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:26.603567 1170 tablet_bootstrap.cc:492] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.604290 749 tablet_service.cc:1468] Processing CreateTablet for tablet 03cf686811e041e7a001669b3d86e28a (DEFAULT_TABLE table=default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.604745 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cf686811e041e7a001669b3d86e28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.604658 1997 consensus_queue.cc:1035] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:26.605374 1170 tablet_bootstrap.cc:654] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.608656 1999 raft_consensus.cc:491] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.608796 1999 raft_consensus.cc:513] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.609383 1999 leader_election.cc:290] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.609964 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.610759 986 leader_election.cc:343] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.611248 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.612980 984 leader_election.cc:343] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.613104 984 leader_election.cc:304] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.613618 1997 raft_consensus.cc:2747] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.616847 1170 tablet_bootstrap.cc:492] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.617074 1171 tablet_bootstrap.cc:492] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.617234 1171 ts_tablet_manager.cc:1397] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.003s
I20250905 19:52:26.617404 1170 ts_tablet_manager.cc:1397] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250905 19:52:26.618007 1171 raft_consensus.cc:357] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.618199 1171 raft_consensus.cc:383] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.618191 1170 raft_consensus.cc:357] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.618297 1171 raft_consensus.cc:738] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.618368 1170 raft_consensus.cc:383] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.618458 1170 raft_consensus.cc:738] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.618502 1171 consensus_queue.cc:260] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.618664 1170 consensus_queue.cc:260] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.619012 1171 ts_tablet_manager.cc:1428] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.619153 1170 ts_tablet_manager.cc:1428] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.619446 1170 tablet_bootstrap.cc:492] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.621171 1171 tablet_bootstrap.cc:492] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.621327 1170 tablet_bootstrap.cc:654] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.624222 1171 tablet_bootstrap.cc:654] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.626355 1997 raft_consensus.cc:491] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.626541 1997 raft_consensus.cc:513] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.627311 1997 leader_election.cc:290] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.627691 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.628264 984 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.629809 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.630978 986 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.631114 986 leader_election.cc:304] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.631780 1997 raft_consensus.cc:2747] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.632159 770 raft_consensus.cc:3058] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.640185 1973 raft_consensus.cc:491] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.640414 1973 raft_consensus.cc:513] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.641232 1973 leader_election.cc:290] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.643373 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.644104 848 leader_election.cc:343] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.645270 1973 consensus_queue.cc:1035] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:26.646169 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.646456 1047 raft_consensus.cc:2466] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.646983 849 leader_election.cc:304] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.647567 1973 raft_consensus.cc:2802] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.647698 1973 raft_consensus.cc:491] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.647797 1973 raft_consensus.cc:3058] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.649057 1171 tablet_bootstrap.cc:492] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.649276 1171 ts_tablet_manager.cc:1397] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.003s
I20250905 19:52:26.650295 1171 raft_consensus.cc:357] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.650496 1171 raft_consensus.cc:383] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.650578 1171 raft_consensus.cc:738] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.650887 1171 consensus_queue.cc:260] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.652845 1171 ts_tablet_manager.cc:1428] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20250905 19:52:26.653272 1171 tablet_bootstrap.cc:492] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.655124 1973 raft_consensus.cc:513] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.655364 1171 tablet_bootstrap.cc:654] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.656447 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.657198 848 leader_election.cc:343] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.658188 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.658411 1048 raft_consensus.cc:3058] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.661713 1048 raft_consensus.cc:2466] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.662426 849 leader_election.cc:304] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.662858 2007 raft_consensus.cc:2802] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.662990 2007 raft_consensus.cc:695] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.663327 2007 consensus_queue.cc:237] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.667284 2095 raft_consensus.cc:491] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.667492 2095 raft_consensus.cc:513] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.669227 2095 leader_election.cc:290] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.670202 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00ac56c449ba4b7b8dbd5e87120799d9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.670967 849 leader_election.cc:343] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9
W20250905 19:52:26.671553 848 leader_election.cc:343] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.671737 848 leader_election.cc:304] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.676079 2095 raft_consensus.cc:2747] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.670171 2092 tablet.cc:1722] T 9f732ae0627c4da5b1691cde9b4d5562 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:26.677302 1973 leader_election.cc:290] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.677641 1171 tablet_bootstrap.cc:492] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.677800 1171 ts_tablet_manager.cc:1397] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.025s user 0.005s sys 0.002s
I20250905 19:52:26.678721 1849 raft_consensus.cc:491] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.678637 1171 raft_consensus.cc:357] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.678849 1171 raft_consensus.cc:383] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.678958 1171 raft_consensus.cc:738] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.678889 1849 raft_consensus.cc:513] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.679219 1171 consensus_queue.cc:260] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.679563 1849 leader_election.cc:290] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.679714 1171 ts_tablet_manager.cc:1428] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:26.680698 1171 tablet_bootstrap.cc:492] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:26.680826 712 leader_election.cc:343] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 36f966bf512d4cbe9ae406386db31533
I20250905 19:52:26.681227 1997 raft_consensus.cc:491] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.681407 1997 raft_consensus.cc:513] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.682327 1997 leader_election.cc:290] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.682422 1171 tablet_bootstrap.cc:654] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.682502 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.683073 984 leader_election.cc:343] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.683465 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.684087 986 leader_election.cc:343] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.684212 986 leader_election.cc:304] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.684820 1997 raft_consensus.cc:2747] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:26.686693 711 leader_election.cc:343] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 36f966bf512d4cbe9ae406386db31533
I20250905 19:52:26.686889 711 leader_election.cc:304] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:26.687222 1849 raft_consensus.cc:2747] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.690116 2095 consensus_queue.cc:1035] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:26.704317 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 117197us
I20250905 19:52:26.708644 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 03cf686811e041e7a001669b3d86e28a (DEFAULT_TABLE table=default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.709160 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 03cf686811e041e7a001669b3d86e28a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.709944 878 tablet_service.cc:1468] Processing CreateTablet for tablet 1bd731a303a545f492b19f774520582c (DEFAULT_TABLE table=default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.710355 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd731a303a545f492b19f774520582c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.712330 749 tablet_service.cc:1468] Processing CreateTablet for tablet 1bd731a303a545f492b19f774520582c (DEFAULT_TABLE table=default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.712746 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd731a303a545f492b19f774520582c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.718859 1584 consensus_queue.cc:1035] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:26.727694 1849 raft_consensus.cc:491] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.727890 1849 raft_consensus.cc:513] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.728730 1849 leader_election.cc:290] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.729735 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0702063ec1c4f6fba58e6d4f3912212" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.729949 913 raft_consensus.cc:2466] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:26.730450 712 leader_election.cc:304] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:26.730863 1849 raft_consensus.cc:2802] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.730978 1849 raft_consensus.cc:491] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.731071 1849 raft_consensus.cc:3058] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.734128 1849 raft_consensus.cc:513] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.735548 1849 leader_election.cc:290] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.735857 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b0702063ec1c4f6fba58e6d4f3912212" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:26.736099 913 raft_consensus.cc:3058] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:26.740613 711 leader_election.cc:343] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212
W20250905 19:52:26.741532 711 leader_election.cc:343] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:26.739928 913 raft_consensus.cc:2466] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:26.743337 712 leader_election.cc:304] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:26.743984 1849 raft_consensus.cc:2802] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.752804 2095 consensus_queue.cc:1035] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:26.775072 1672 raft_consensus.cc:491] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.775372 1672 raft_consensus.cc:513] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.776840 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.777491 848 leader_election.cc:343] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.778689 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.778978 1045 raft_consensus.cc:2466] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.779457 849 leader_election.cc:304] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.779847 1658 raft_consensus.cc:2802] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.780066 1658 raft_consensus.cc:491] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.780226 1658 raft_consensus.cc:3058] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.783565 1658 raft_consensus.cc:513] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.785838 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.786427 848 leader_election.cc:343] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.787926 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 1bd731a303a545f492b19f774520582c (DEFAULT_TABLE table=default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.788401 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1bd731a303a545f492b19f774520582c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.788682 878 tablet_service.cc:1468] Processing CreateTablet for tablet 7e8bfe047a6e4cc087eea75244da5b41 (DEFAULT_TABLE table=default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.789070 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e8bfe047a6e4cc087eea75244da5b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.792717 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7e8bfe047a6e4cc087eea75244da5b41 (DEFAULT_TABLE table=default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.793145 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e8bfe047a6e4cc087eea75244da5b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.800036 1997 raft_consensus.cc:491] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.800182 1997 raft_consensus.cc:513] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.800774 1997 leader_election.cc:290] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.801271 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8b6823580cb4040b49ed0a49b748965" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.801777 984 leader_election.cc:343] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.802235 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8b6823580cb4040b49ed0a49b748965" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.802706 986 leader_election.cc:343] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.802814 986 leader_election.cc:304] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.803228 1997 raft_consensus.cc:2747] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.809149 1171 tablet_bootstrap.cc:492] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.809341 1171 ts_tablet_manager.cc:1397] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.129s user 0.034s sys 0.036s
I20250905 19:52:26.810227 1171 raft_consensus.cc:357] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.810488 1171 raft_consensus.cc:383] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.810570 1171 raft_consensus.cc:738] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.810813 1171 consensus_queue.cc:260] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.811383 1171 ts_tablet_manager.cc:1428] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.811722 1171 tablet_bootstrap.cc:492] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.818799 1849 raft_consensus.cc:695] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:26.819202 1171 tablet_bootstrap.cc:654] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.819252 1849 consensus_queue.cc:237] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:26.822412 848 consensus_peers.cc:489] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:26.822587 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:26.822957 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:26.823585 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:26.823838 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:26.824112 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:26.824445 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:26.824594 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:26.824806 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:26.825249 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:26.825603 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:26.826040 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:26.826323 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:26.826776 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:26.827345 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:26.827579 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:26.828445 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac]). Expected version 2 got 0
I20250905 19:52:26.828824 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:26.829123 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:26.829072 1672 leader_election.cc:290] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.829319 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:26.829680 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:26.829876 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:26.830085 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_849b4e364b3f4ff2b47a2347f84d6b0d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 1 got 0
I20250905 19:52:26.830273 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:26.830484 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:26.830686 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:26.831183 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:26.831858 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_9da2eedb50e64356ba6a49d93bafbe28 [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 1 got 0
I20250905 19:52:26.832087 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:26.832330 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:26.832527 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:26.832696 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:26.832870 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:26.835845 1170 tablet_bootstrap.cc:492] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.836057 1170 ts_tablet_manager.cc:1397] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.217s user 0.013s sys 0.041s
I20250905 19:52:26.836833 1170 raft_consensus.cc:357] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.837057 1170 raft_consensus.cc:383] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.837154 1170 raft_consensus.cc:738] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.837391 1170 consensus_queue.cc:260] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.848726 1169 tablet_bootstrap.cc:492] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.848872 1169 ts_tablet_manager.cc:1397] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.289s user 0.045s sys 0.050s
I20250905 19:52:26.849624 1169 raft_consensus.cc:357] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.849792 1169 raft_consensus.cc:383] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.849960 1169 raft_consensus.cc:738] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.850142 1169 consensus_queue.cc:260] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.850836 1169 ts_tablet_manager.cc:1428] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:26.851235 1169 tablet_bootstrap.cc:492] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.852947 1169 tablet_bootstrap.cc:654] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.837885 1170 ts_tablet_manager.cc:1428] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.857036 1170 tablet_bootstrap.cc:492] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:26.860368 1997 raft_consensus.cc:491] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.860589 1997 raft_consensus.cc:513] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.861581 1798 raft_consensus.cc:491] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.861707 1798 raft_consensus.cc:513] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.862179 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e15350150db413fa4f100171280f46e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.862295 1798 leader_election.cc:290] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:26.862643 984 leader_election.cc:343] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.862927 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:26.863390 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.863898 1997 leader_election.cc:290] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:26.863938 984 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.864154 986 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.864243 986 leader_election.cc:304] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.864615 1997 raft_consensus.cc:2747] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.864627 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e15350150db413fa4f100171280f46e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:26.865190 986 leader_election.cc:343] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.865289 986 leader_election.cc:304] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.866773 1169 tablet_bootstrap.cc:492] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.866906 1169 ts_tablet_manager.cc:1397] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.016s user 0.000s sys 0.003s
W20250905 19:52:26.867215 848 consensus_peers.cc:489] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:26.867442 1997 raft_consensus.cc:2747] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.868360 1169 raft_consensus.cc:357] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.868536 1169 raft_consensus.cc:383] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.868600 1169 raft_consensus.cc:738] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.868786 1170 tablet_bootstrap.cc:654] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.868880 1169 consensus_queue.cc:260] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.871188 1169 ts_tablet_manager.cc:1428] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250905 19:52:26.872104 1169 tablet_bootstrap.cc:492] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.872468 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 7e8bfe047a6e4cc087eea75244da5b41 (DEFAULT_TABLE table=default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.872830 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e8bfe047a6e4cc087eea75244da5b41. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.873396 1171 tablet_bootstrap.cc:492] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.873567 1171 ts_tablet_manager.cc:1397] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.062s user 0.004s sys 0.000s
I20250905 19:52:26.873752 1169 tablet_bootstrap.cc:654] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.874513 1171 raft_consensus.cc:357] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.874729 1171 raft_consensus.cc:383] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.874790 1171 raft_consensus.cc:738] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.874979 1171 consensus_queue.cc:260] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.875419 1171 ts_tablet_manager.cc:1428] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.879156 1171 tablet_bootstrap.cc:492] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.881693 1169 tablet_bootstrap.cc:492] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.881841 1169 ts_tablet_manager.cc:1397] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20250905 19:52:26.883001 2108 raft_consensus.cc:491] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.883154 2108 raft_consensus.cc:513] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.883737 2108 leader_election.cc:290] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.884490 1171 tablet_bootstrap.cc:654] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.885008 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.885644 848 leader_election.cc:343] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.884622 1169 raft_consensus.cc:357] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.886909 1169 raft_consensus.cc:383] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.886986 1169 raft_consensus.cc:738] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.887197 1169 consensus_queue.cc:260] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.888059 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:26.888291 1048 raft_consensus.cc:2466] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:26.889025 849 leader_election.cc:304] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.889747 1169 ts_tablet_manager.cc:1428] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 19:52:26.890856 1169 tablet_bootstrap.cc:492] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.893357 749 tablet_service.cc:1468] Processing CreateTablet for tablet c8e53152ada84954b12bbfe9d1fe3c16 (DEFAULT_TABLE table=default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.893831 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e53152ada84954b12bbfe9d1fe3c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.894451 1022 tablet_service.cc:1468] Processing CreateTablet for tablet c8e53152ada84954b12bbfe9d1fe3c16 (DEFAULT_TABLE table=default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.894891 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e53152ada84954b12bbfe9d1fe3c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.896181 2108 raft_consensus.cc:2802] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:26.896274 2108 raft_consensus.cc:491] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:26.896340 2108 raft_consensus.cc:3058] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.896296 891 tablet_service.cc:1468] Processing CreateTablet for tablet c8e53152ada84954b12bbfe9d1fe3c16 (DEFAULT_TABLE table=default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.896656 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e53152ada84954b12bbfe9d1fe3c16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.898918 2108 raft_consensus.cc:513] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.899662 2108 leader_election.cc:290] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.901012 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:26.901577 848 consensus_peers.cc:489] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:26.901728 848 leader_election.cc:343] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.903656 754 tablet_service.cc:1468] Processing CreateTablet for tablet fbba1ad28a614b06afe865c478901a32 (DEFAULT_TABLE table=default.Table_6ec4f72266c7447da59b8544cfa98cd6 [id=0b6f3d59aebb4de0b34652b9f2c8ded7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.904059 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbba1ad28a614b06afe865c478901a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.908227 1997 raft_consensus.cc:491] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.908389 1997 raft_consensus.cc:513] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.909704 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c46ea498d6a14d1ebdffea415cfc7114" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.910281 984 leader_election.cc:343] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.911080 1997 leader_election.cc:290] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:26.911080 2109 raft_consensus.cc:491] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.911209 2109 raft_consensus.cc:513] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.911381 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c46ea498d6a14d1ebdffea415cfc7114" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:26.911794 2109 leader_election.cc:290] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:26.911932 986 leader_election.cc:343] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.912050 986 leader_election.cc:304] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.912499 1997 raft_consensus.cc:2747] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.914610 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13d282e1dc54d1faa2882c9b39d17cb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.915740 848 leader_election.cc:343] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:26.919828 2109 raft_consensus.cc:491] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.919991 2109 raft_consensus.cc:513] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.920559 2109 leader_election.cc:290] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.920708 891 tablet_service.cc:1468] Processing CreateTablet for tablet fbba1ad28a614b06afe865c478901a32 (DEFAULT_TABLE table=default.Table_6ec4f72266c7447da59b8544cfa98cd6 [id=0b6f3d59aebb4de0b34652b9f2c8ded7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.921104 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbba1ad28a614b06afe865c478901a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.921406 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9630f1feb6d74472b5f6f4b54151311c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.921970 848 leader_election.cc:343] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.923209 849 leader_election.cc:343] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:26.923334 849 leader_election.cc:304] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.923571 2109 raft_consensus.cc:2747] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:26.924655 849 leader_election.cc:343] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:26.924814 849 leader_election.cc:304] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.925074 2109 raft_consensus.cc:2747] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.925981 2109 consensus_queue.cc:1035] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:26.928720 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.929047 1048 raft_consensus.cc:3058] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.930542 1028 tablet_service.cc:1468] Processing CreateTablet for tablet fbba1ad28a614b06afe865c478901a32 (DEFAULT_TABLE table=default.Table_6ec4f72266c7447da59b8544cfa98cd6 [id=0b6f3d59aebb4de0b34652b9f2c8ded7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.930935 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fbba1ad28a614b06afe865c478901a32. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.932293 1169 tablet_bootstrap.cc:654] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.934887 1048 raft_consensus.cc:2466] T de19c897d3314fecb17cedb3370eed93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.935079 1169 tablet_bootstrap.cc:492] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.935215 1169 ts_tablet_manager.cc:1397] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20250905 19:52:26.935534 849 leader_election.cc:304] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.936082 2007 raft_consensus.cc:2802] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.935986 1169 raft_consensus.cc:357] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.936190 1169 raft_consensus.cc:383] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.936276 1169 raft_consensus.cc:738] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.936436 2007 raft_consensus.cc:695] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.936523 1169 consensus_queue.cc:260] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.937700 2007 consensus_queue.cc:237] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.939265 1169 ts_tablet_manager.cc:1428] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:26.939604 1169 tablet_bootstrap.cc:492] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.941385 1169 tablet_bootstrap.cc:654] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.952986 1169 tablet_bootstrap.cc:492] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
W20250905 19:52:26.953012 848 consensus_peers.cc:489] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:26.953184 1169 ts_tablet_manager.cc:1397] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250905 19:52:26.954077 1169 raft_consensus.cc:357] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.954294 1169 raft_consensus.cc:383] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.954352 1169 raft_consensus.cc:738] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.954555 1169 consensus_queue.cc:260] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.955163 1169 ts_tablet_manager.cc:1428] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.955420 1169 tablet_bootstrap.cc:492] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.956625 1658 leader_election.cc:290] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.957086 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:26.957322 1048 raft_consensus.cc:3058] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:26.958285 749 tablet_service.cc:1468] Processing CreateTablet for tablet 414f30a772524c33a74864ff741c865f (DEFAULT_TABLE table=default.Table_a3f620901eff44dea30bf9e4de3c5552 [id=93e1f25857f644c5bd22e445576390a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.959322 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 414f30a772524c33a74864ff741c865f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.960170 1048 raft_consensus.cc:2466] T 9b3acf17eab64913b5b33677196a5781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:26.960650 849 leader_election.cc:304] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.964159 2007 raft_consensus.cc:2802] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:26.964382 2007 raft_consensus.cc:695] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:26.964709 2007 consensus_queue.cc:237] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.965408 1169 tablet_bootstrap.cc:654] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.965930 877 tablet_service.cc:1468] Processing CreateTablet for tablet 414f30a772524c33a74864ff741c865f (DEFAULT_TABLE table=default.Table_a3f620901eff44dea30bf9e4de3c5552 [id=93e1f25857f644c5bd22e445576390a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.966341 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 414f30a772524c33a74864ff741c865f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.966545 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 414f30a772524c33a74864ff741c865f (DEFAULT_TABLE table=default.Table_a3f620901eff44dea30bf9e4de3c5552 [id=93e1f25857f644c5bd22e445576390a7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.966938 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 414f30a772524c33a74864ff741c865f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.970126 1169 tablet_bootstrap.cc:492] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.970338 1169 ts_tablet_manager.cc:1397] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250905 19:52:26.972316 2095 raft_consensus.cc:491] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:26.972462 2095 raft_consensus.cc:513] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.973049 2095 leader_election.cc:290] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:26.973238 1169 raft_consensus.cc:357] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.973510 1169 raft_consensus.cc:383] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.973637 1169 raft_consensus.cc:738] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.973769 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f3473785fff45d692bd50b1015ffe39" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:26.974395 848 leader_election.cc:343] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:26.975688 849 leader_election.cc:343] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:26.975849 849 leader_election.cc:304] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:26.976276 1672 raft_consensus.cc:2747] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:26.976465 1169 consensus_queue.cc:260] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.976998 1169 ts_tablet_manager.cc:1428] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:26.977298 1169 tablet_bootstrap.cc:492] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:26.978962 1169 tablet_bootstrap.cc:654] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:26.984241 749 tablet_service.cc:1468] Processing CreateTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f (DEFAULT_TABLE table=default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.984836 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb9379335ca4eb5a32a6cbe85601a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.987777 1169 tablet_bootstrap.cc:492] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:26.987990 1169 ts_tablet_manager.cc:1397] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:26.988901 1169 raft_consensus.cc:357] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.989109 1169 raft_consensus.cc:383] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.989187 1169 raft_consensus.cc:738] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.995736 1171 tablet_bootstrap.cc:492] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:26.995899 1171 ts_tablet_manager.cc:1397] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.117s user 0.017s sys 0.016s
I20250905 19:52:26.995738 891 tablet_service.cc:1468] Processing CreateTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f (DEFAULT_TABLE table=default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.996680 1171 raft_consensus.cc:357] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.996837 1170 tablet_bootstrap.cc:492] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:26.996870 1171 raft_consensus.cc:383] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.996927 1171 raft_consensus.cc:738] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.997124 1170 ts_tablet_manager.cc:1397] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.141s user 0.019s sys 0.030s
I20250905 19:52:26.997149 1171 consensus_queue.cc:260] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.997633 1171 ts_tablet_manager.cc:1428] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:26.997915 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef9c9c6f81c4daea9008ada454b0659 (DEFAULT_TABLE table=default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:26.998114 1171 tablet_bootstrap.cc:492] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:26.998210 1170 raft_consensus.cc:357] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.998485 1170 raft_consensus.cc:383] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:26.998612 1170 raft_consensus.cc:738] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:26.998929 1170 consensus_queue.cc:260] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:26.999424 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef9c9c6f81c4daea9008ada454b0659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:26.996845 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb9379335ca4eb5a32a6cbe85601a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.003561 1658 raft_consensus.cc:491] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.003695 1658 raft_consensus.cc:513] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.003856 1171 tablet_bootstrap.cc:654] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.004910 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e9994863c374354838aa0f817d0057e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.005131 1048 raft_consensus.cc:2466] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.005479 877 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef9c9c6f81c4daea9008ada454b0659 (DEFAULT_TABLE table=default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.005616 849 leader_election.cc:304] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.005892 2007 raft_consensus.cc:2802] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.005987 2007 raft_consensus.cc:491] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.006049 2007 raft_consensus.cc:3058] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.006239 1022 tablet_service.cc:1468] Processing CreateTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f (DEFAULT_TABLE table=default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.006596 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet edb9379335ca4eb5a32a6cbe85601a1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.008124 1658 leader_election.cc:290] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.009032 2007 raft_consensus.cc:513] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.009678 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e9994863c374354838aa0f817d0057e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.010020 1169 consensus_queue.cc:260] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.010486 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e9994863c374354838aa0f817d0057e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.010675 1048 raft_consensus.cc:3058] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.010840 1169 ts_tablet_manager.cc:1428] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.023s user 0.001s sys 0.000s
I20250905 19:52:27.011271 1169 tablet_bootstrap.cc:492] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.013446 1048 raft_consensus.cc:2466] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.013648 1169 tablet_bootstrap.cc:654] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.014633 849 leader_election.cc:304] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.015347 1170 ts_tablet_manager.cc:1428] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250905 19:52:27.015722 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9e9994863c374354838aa0f817d0057e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.015249 1658 raft_consensus.cc:2802] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.016278 1658 raft_consensus.cc:695] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.016593 1658 consensus_queue.cc:237] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.016917 848 leader_election.cc:343] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.017900 848 leader_election.cc:343] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.018432 1170 tablet_bootstrap.cc:492] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.019476 2007 leader_election.cc:290] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.020359 1170 tablet_bootstrap.cc:654] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.005903 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef9c9c6f81c4daea9008ada454b0659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.022990 1997 raft_consensus.cc:491] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.023165 1997 raft_consensus.cc:513] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.023787 1997 leader_election.cc:290] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.024377 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.024762 878 tablet_service.cc:1468] Processing CreateTablet for tablet 6ee9f4fe3a464abaaca48f22ea183c0e (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:27.024885 986 leader_election.cc:343] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.025121 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee9f4fe3a464abaaca48f22ea183c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.025750 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 7ef9c9c6f81c4daea9008ada454b0659 (DEFAULT_TABLE table=default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.026216 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7ef9c9c6f81c4daea9008ada454b0659. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.032229 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 6ee9f4fe3a464abaaca48f22ea183c0e (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.032648 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee9f4fe3a464abaaca48f22ea183c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.032974 754 tablet_service.cc:1468] Processing CreateTablet for tablet 6ee9f4fe3a464abaaca48f22ea183c0e (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.033340 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6ee9f4fe3a464abaaca48f22ea183c0e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.034224 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.034986 984 leader_election.cc:343] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.035141 984 leader_election.cc:304] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.035668 1798 raft_consensus.cc:2747] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.036355 773 raft_consensus.cc:3058] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.041535 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:27.041831 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:27.042349 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:27.042573 1532 catalog_manager.cc:5582] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.043644 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:27.072963 1170 tablet_bootstrap.cc:492] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:27.073164 1170 ts_tablet_manager.cc:1397] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.056s user 0.003s sys 0.000s
I20250905 19:52:27.073956 1170 raft_consensus.cc:357] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.074180 1170 raft_consensus.cc:383] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.074267 1170 raft_consensus.cc:738] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.074527 1170 consensus_queue.cc:260] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.077152 1170 ts_tablet_manager.cc:1428] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:27.077518 1170 tablet_bootstrap.cc:492] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.080147 1997 raft_consensus.cc:491] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.080339 1997 raft_consensus.cc:513] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.081028 1997 leader_election.cc:290] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.081965 2126 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:27.082039 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.082147 2126 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:27.082216 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:27.082815 986 leader_election.cc:343] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.083009 2126 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
W20250905 19:52:27.083091 984 leader_election.cc:343] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.083218 984 leader_election.cc:304] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.083690 1997 raft_consensus.cc:2747] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.083837 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 615 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:27.043846 1532 catalog_manager.cc:5582] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.086534 1532 catalog_manager.cc:5582] T 7cdd4de055864d4fa6adb36bf190e609 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.089759 1170 tablet_bootstrap.cc:654] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.097846 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:27.098145 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:27.098656 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:27.099692 1169 tablet_bootstrap.cc:492] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.099886 1169 ts_tablet_manager.cc:1397] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.089s user 0.005s sys 0.016s
I20250905 19:52:27.100879 1169 raft_consensus.cc:357] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.101086 1169 raft_consensus.cc:383] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.101186 1169 raft_consensus.cc:738] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.101536 1532 catalog_manager.cc:5582] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.103241 1171 tablet_bootstrap.cc:492] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.102097 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:27.103387 1171 ts_tablet_manager.cc:1397] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.105s user 0.027s sys 0.014s
I20250905 19:52:27.103899 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:27.104128 1171 raft_consensus.cc:357] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.104225 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:27.104326 1171 raft_consensus.cc:383] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.104385 1171 raft_consensus.cc:738] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.104585 1171 consensus_queue.cc:260] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.104739 1169 consensus_queue.cc:260] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.105046 1171 ts_tablet_manager.cc:1428] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:27.105335 1171 tablet_bootstrap.cc:492] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.106988 1171 tablet_bootstrap.cc:654] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.116580 1169 ts_tablet_manager.cc:1428] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250905 19:52:27.120236 1849 raft_consensus.cc:491] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.120522 1849 raft_consensus.cc:513] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.121325 1849 leader_election.cc:290] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.122334 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.123647 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7cdd4de055864d4fa6adb36bf190e609" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.124697 711 leader_election.cc:304] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:27.125084 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:27.125371 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:27.125914 1169 tablet_bootstrap.cc:492] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.126693 1849 raft_consensus.cc:2747] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.128376 1169 tablet_bootstrap.cc:654] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:27.129127 511 leader_election.cc:336] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection refused (error 111)
I20250905 19:52:27.129356 511 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:27.129537 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:27.129729 2126 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:52:27.130290 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:27.130586 1532 catalog_manager.cc:5582] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.131069 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:27.131628 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:27.132961 1170 tablet_bootstrap.cc:492] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:27.133128 1170 ts_tablet_manager.cc:1397] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.056s user 0.009s sys 0.006s
I20250905 19:52:27.132858 1532 catalog_manager.cc:5582] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.133807 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:27.134231 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:27.134510 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:27.134797 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_849b4e364b3f4ff2b47a2347f84d6b0d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 1 got 0
I20250905 19:52:27.135200 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:27.135519 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:27.135543 1170 raft_consensus.cc:357] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.135747 1170 raft_consensus.cc:383] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.135848 1170 raft_consensus.cc:738] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.136029 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:27.136780 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:27.137363 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:27.137534 1171 tablet_bootstrap.cc:492] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.137682 1171 ts_tablet_manager.cc:1397] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.032s user 0.002s sys 0.001s
I20250905 19:52:27.137986 1532 catalog_manager.cc:5582] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.138442 1171 raft_consensus.cc:357] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.138614 1171 raft_consensus.cc:383] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.138670 1171 raft_consensus.cc:738] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.138881 1171 consensus_queue.cc:260] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.138978 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_9da2eedb50e64356ba6a49d93bafbe28 [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 1 got 0
I20250905 19:52:27.139364 1171 ts_tablet_manager.cc:1428] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:27.139624 1171 tablet_bootstrap.cc:492] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.139595 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:27.140218 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:27.140674 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:27.141281 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:27.141425 1171 tablet_bootstrap.cc:654] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.141995 1532 catalog_manager.cc:5582] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:27.143280 1170 consensus_queue.cc:260] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.143448 1532 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:27.148789 1169 tablet_bootstrap.cc:492] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.149092 1169 ts_tablet_manager.cc:1397] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20250905 19:52:27.150168 1169 raft_consensus.cc:357] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.150458 1169 raft_consensus.cc:383] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.150583 1169 raft_consensus.cc:738] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.151005 1169 consensus_queue.cc:260] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.153286 1170 ts_tablet_manager.cc:1428] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.020s user 0.003s sys 0.000s
W20250905 19:52:27.155306 848 consensus_peers.cc:489] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.156791 1849 raft_consensus.cc:491] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.156978 1849 raft_consensus.cc:513] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.157648 1849 leader_election.cc:290] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.158377 712 leader_election.cc:343] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 29c2b030fafd44ad9e648b2527a4ad07
W20250905 19:52:27.158533 711 leader_election.cc:343] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 29c2b030fafd44ad9e648b2527a4ad07
I20250905 19:52:27.158679 711 leader_election.cc:304] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:27.159219 1849 raft_consensus.cc:2747] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.159564 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:27.160746 984 consensus_peers.cc:489] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.162675 1169 ts_tablet_manager.cc:1428] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250905 19:52:27.163476 1170 tablet_bootstrap.cc:492] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.165573 1170 tablet_bootstrap.cc:654] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.168474 1973 raft_consensus.cc:491] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.168637 1973 raft_consensus.cc:513] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.169322 1973 leader_election.cc:290] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.170552 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.170783 1048 raft_consensus.cc:2466] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.171237 849 leader_election.cc:304] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.171936 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.172482 1973 raft_consensus.cc:2802] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.172566 1973 raft_consensus.cc:491] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.172622 1973 raft_consensus.cc:3058] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.175302 1973 raft_consensus.cc:513] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.176548 848 leader_election.cc:343] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.177000 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.178000 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.178189 1048 raft_consensus.cc:3058] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.181252 1048 raft_consensus.cc:2466] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.181918 849 leader_election.cc:304] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.182260 2007 raft_consensus.cc:2802] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.182397 2007 raft_consensus.cc:695] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.182626 2007 consensus_queue.cc:237] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.182700 848 leader_election.cc:343] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.183095 1973 leader_election.cc:290] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.185379 1169 tablet_bootstrap.cc:492] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.187585 1169 tablet_bootstrap.cc:654] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.196704 1171 tablet_bootstrap.cc:492] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.196925 1171 ts_tablet_manager.cc:1397] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.057s user 0.019s sys 0.000s
I20250905 19:52:27.197876 1171 raft_consensus.cc:357] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.198076 1171 raft_consensus.cc:383] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.198148 1171 raft_consensus.cc:738] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.198371 1171 consensus_queue.cc:260] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.198943 1171 ts_tablet_manager.cc:1428] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:27.199672 1171 tablet_bootstrap.cc:492] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.201642 1171 tablet_bootstrap.cc:654] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.202121 1169 tablet_bootstrap.cc:492] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.202536 1169 ts_tablet_manager.cc:1397] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250905 19:52:27.203743 1169 raft_consensus.cc:357] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.204005 1169 raft_consensus.cc:383] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.204100 1169 raft_consensus.cc:738] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.205618 1169 consensus_queue.cc:260] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.210240 1171 tablet_bootstrap.cc:492] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.210387 1171 ts_tablet_manager.cc:1397] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.000s
I20250905 19:52:27.211158 1169 ts_tablet_manager.cc:1428] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250905 19:52:27.211143 1171 raft_consensus.cc:357] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.211293 1171 raft_consensus.cc:383] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.211338 1171 raft_consensus.cc:738] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.211540 1171 consensus_queue.cc:260] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.211683 1169 tablet_bootstrap.cc:492] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.211956 1171 ts_tablet_manager.cc:1428] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250905 19:52:27.212235 1171 tablet_bootstrap.cc:492] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.213668 1171 tablet_bootstrap.cc:654] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.213898 1169 tablet_bootstrap.cc:654] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.215657 1170 tablet_bootstrap.cc:492] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:27.216302 1170 ts_tablet_manager.cc:1397] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.053s user 0.006s sys 0.004s
I20250905 19:52:27.217216 1170 raft_consensus.cc:357] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.224041 1170 raft_consensus.cc:383] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.224239 1170 raft_consensus.cc:738] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.224674 1170 consensus_queue.cc:260] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.225564 1170 ts_tablet_manager.cc:1428] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20250905 19:52:27.226336 1170 tablet_bootstrap.cc:492] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.229058 1849 raft_consensus.cc:491] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.229296 1849 raft_consensus.cc:513] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.230023 1849 leader_election.cc:290] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.230948 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "302053e1611b4e22a9a2e7b60ae68f27" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.231138 913 raft_consensus.cc:2466] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
W20250905 19:52:27.232173 711 leader_election.cc:343] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:27.232681 712 leader_election.cc:304] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.233158 1849 raft_consensus.cc:2802] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.233251 1849 raft_consensus.cc:491] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.233341 1849 raft_consensus.cc:3058] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.228856 1170 tablet_bootstrap.cc:654] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.239988 1169 tablet_bootstrap.cc:492] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.240187 1169 ts_tablet_manager.cc:1397] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.005s
I20250905 19:52:27.240988 1169 raft_consensus.cc:357] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.241184 1169 raft_consensus.cc:383] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.241282 1169 raft_consensus.cc:738] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.241547 1849 raft_consensus.cc:513] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.241544 1169 consensus_queue.cc:260] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.242225 1849 leader_election.cc:290] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.243708 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "302053e1611b4e22a9a2e7b60ae68f27" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.243884 913 raft_consensus.cc:3058] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.244549 1169 ts_tablet_manager.cc:1428] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:27.244846 1169 tablet_bootstrap.cc:492] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.246447 1169 tablet_bootstrap.cc:654] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.246562 913 raft_consensus.cc:2466] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:27.249897 712 leader_election.cc:304] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:27.252480 1171 tablet_bootstrap.cc:492] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.252662 1171 ts_tablet_manager.cc:1397] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.000s
I20250905 19:52:27.253665 1171 raft_consensus.cc:357] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.253901 1171 raft_consensus.cc:383] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.253971 1171 raft_consensus.cc:738] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.254199 2007 raft_consensus.cc:491] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.254386 2007 raft_consensus.cc:513] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.255163 2007 leader_election.cc:290] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.256639 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3805127dd78489e8545da7414ead904" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.256659 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3805127dd78489e8545da7414ead904" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.256867 1048 raft_consensus.cc:2466] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.257297 849 leader_election.cc:304] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.257557 2007 raft_consensus.cc:2802] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.257624 2007 raft_consensus.cc:491] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.257690 2007 raft_consensus.cc:3058] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:27.258301 711 leader_election.cc:343] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27
W20250905 19:52:27.258680 848 leader_election.cc:343] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.254227 1171 consensus_queue.cc:260] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.259191 1849 raft_consensus.cc:2802] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.259361 1171 ts_tablet_manager.cc:1428] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:27.259622 1171 tablet_bootstrap.cc:492] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.260100 1849 raft_consensus.cc:695] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:27.260383 1849 consensus_queue.cc:237] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.261713 1171 tablet_bootstrap.cc:654] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.265027 1171 tablet_bootstrap.cc:492] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.265161 1171 ts_tablet_manager.cc:1397] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:27.265815 1171 raft_consensus.cc:357] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.265990 1171 raft_consensus.cc:383] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.266050 1171 raft_consensus.cc:738] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.270100 1171 consensus_queue.cc:260] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.270709 1171 ts_tablet_manager.cc:1428] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:27.271076 1171 tablet_bootstrap.cc:492] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.272903 1171 tablet_bootstrap.cc:654] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.276906 2007 raft_consensus.cc:513] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.278012 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3805127dd78489e8545da7414ead904" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.278264 1048 raft_consensus.cc:3058] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.281713 1048 raft_consensus.cc:2466] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.282243 849 leader_election.cc:304] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.282582 1658 raft_consensus.cc:2802] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.282712 1658 raft_consensus.cc:695] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.282982 1658 consensus_queue.cc:237] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.284775 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3805127dd78489e8545da7414ead904" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.285398 848 leader_election.cc:343] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.286798 2100 raft_consensus.cc:491] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.286976 2100 raft_consensus.cc:513] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.288336 2100 leader_election.cc:290] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.288424 1973 raft_consensus.cc:491] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.288558 1973 raft_consensus.cc:513] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.289098 1973 leader_election.cc:290] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.289363 711 leader_election.cc:343] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:27.289868 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03a12cf719f34096bb82f5846bbe3b2d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.290513 848 leader_election.cc:343] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.290963 1973 raft_consensus.cc:491] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.291091 1973 raft_consensus.cc:513] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.291256 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2ed1bc09c94526bc9662f2d4d21f8f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.291416 913 raft_consensus.cc:2466] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:27.291584 1973 leader_election.cc:290] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.291839 849 leader_election.cc:343] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:27.291989 849 leader_election.cc:304] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.292234 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723168b075874dfcb3eb50c5134c499e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.292451 712 leader_election.cc:304] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.292776 1171 tablet_bootstrap.cc:492] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.292812 2100 raft_consensus.cc:2802] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.292830 1973 raft_consensus.cc:2747] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.292920 2100 raft_consensus.cc:491] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.292953 1171 ts_tablet_manager.cc:1397] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.004s sys 0.000s
I20250905 19:52:27.293005 2100 raft_consensus.cc:3058] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.293308 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723168b075874dfcb3eb50c5134c499e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.293495 1050 raft_consensus.cc:2466] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.293874 849 leader_election.cc:304] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.293810 1171 raft_consensus.cc:357] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.294008 1171 raft_consensus.cc:383] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.294070 1171 raft_consensus.cc:738] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.294128 1973 raft_consensus.cc:2802] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.294189 1973 raft_consensus.cc:491] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.294246 1973 raft_consensus.cc:3058] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.294317 1171 consensus_queue.cc:260] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.295034 1171 ts_tablet_manager.cc:1428] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:27.295281 1171 tablet_bootstrap.cc:492] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.295790 2100 raft_consensus.cc:513] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.296644 848 leader_election.cc:343] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.296906 1171 tablet_bootstrap.cc:654] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.296867 1973 raft_consensus.cc:513] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.297396 1973 leader_election.cc:290] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.297782 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723168b075874dfcb3eb50c5134c499e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.298333 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "723168b075874dfcb3eb50c5134c499e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.298524 1050 raft_consensus.cc:3058] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:27.298583 848 leader_election.cc:343] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.298964 1171 tablet_bootstrap.cc:492] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.299085 1171 ts_tablet_manager.cc:1397] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20250905 19:52:27.299816 1171 raft_consensus.cc:357] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.300007 1171 raft_consensus.cc:383] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.300079 1171 raft_consensus.cc:738] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.300297 1171 consensus_queue.cc:260] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.301302 1171 ts_tablet_manager.cc:1428] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:27.301802 1050 raft_consensus.cc:2466] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.301892 1171 tablet_bootstrap.cc:492] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:27.302284 711 leader_election.cc:343] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:27.302686 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d2ed1bc09c94526bc9662f2d4d21f8f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.302850 913 raft_consensus.cc:3058] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.303804 1171 tablet_bootstrap.cc:654] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.305402 913 raft_consensus.cc:2466] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:27.305773 712 leader_election.cc:304] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.306017 1997 consensus_queue.cc:1035] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.308781 1672 consensus_queue.cc:1035] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.320097 2100 leader_election.cc:290] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.320451 2100 raft_consensus.cc:2802] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.320595 2100 raft_consensus.cc:695] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:27.320952 2100 consensus_queue.cc:237] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.323894 2007 leader_election.cc:290] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.325136 2007 raft_consensus.cc:491] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.325140 1997 raft_consensus.cc:491] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.325302 2007 raft_consensus.cc:513] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.325340 1997 raft_consensus.cc:513] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.326072 1997 leader_election.cc:290] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.326568 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfd03ad0bd34b6ca86e7b07a62b2a8e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.327275 848 leader_election.cc:343] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.327412 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f1a8e7059b348cba34069ef49bf1aad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.327585 2007 leader_election.cc:290] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.327580 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f1a8e7059b348cba34069ef49bf1aad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.327610 913 raft_consensus.cc:2466] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:27.328176 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfd03ad0bd34b6ca86e7b07a62b2a8e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:27.328285 984 leader_election.cc:343] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.328446 1048 raft_consensus.cc:2466] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.328591 986 leader_election.cc:304] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.328940 849 leader_election.cc:304] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.329047 1997 raft_consensus.cc:2802] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.329110 1997 raft_consensus.cc:491] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.329169 1997 raft_consensus.cc:3058] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.331389 2007 raft_consensus.cc:2802] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.331470 2007 raft_consensus.cc:491] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.331522 2007 raft_consensus.cc:3058] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.332504 1997 raft_consensus.cc:513] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.333461 1973 raft_consensus.cc:491] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.333614 1973 raft_consensus.cc:513] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.333696 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f1a8e7059b348cba34069ef49bf1aad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.333874 913 raft_consensus.cc:3058] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.334441 1973 leader_election.cc:290] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.334954 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44769f1170354870ae2d34fdb67ffbae" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.335480 848 leader_election.cc:343] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.336925 913 raft_consensus.cc:2466] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:27.338346 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f1a8e7059b348cba34069ef49bf1aad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.333806 1997 leader_election.cc:290] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.337600 986 leader_election.cc:304] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.337671 1798 raft_consensus.cc:491] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.339913 1798 raft_consensus.cc:513] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.340641 1798 leader_election.cc:290] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.340794 1798 raft_consensus.cc:2802] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.340898 1798 raft_consensus.cc:695] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:27.341213 1798 consensus_queue.cc:237] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.342409 1973 raft_consensus.cc:491] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.342558 1973 raft_consensus.cc:513] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.342927 984 leader_election.cc:343] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.343722 1973 leader_election.cc:290] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.343715 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab4f69c45295422f85b310aa55220e9b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.343995 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.344285 984 leader_election.cc:343] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.344401 848 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.344923 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab4f69c45295422f85b310aa55220e9b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.345098 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.345286 1046 raft_consensus.cc:2466] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.345494 913 raft_consensus.cc:2466] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:27.345815 986 leader_election.cc:304] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.345857 849 leader_election.cc:304] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.336072 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44769f1170354870ae2d34fdb67ffbae" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.346318 1048 raft_consensus.cc:2466] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.346702 849 leader_election.cc:304] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.346935 1672 raft_consensus.cc:2802] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.347016 1672 raft_consensus.cc:491] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.347033 1973 raft_consensus.cc:2802] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.347082 1672 raft_consensus.cc:3058] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.350106 1997 raft_consensus.cc:2802] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.350284 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:27.350275 1997 raft_consensus.cc:491] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.350358 1997 raft_consensus.cc:3058] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.353003 1997 raft_consensus.cc:513] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.347084 1973 raft_consensus.cc:491] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.353730 1973 raft_consensus.cc:3058] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.357230 1672 raft_consensus.cc:513] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.357870 1672 leader_election.cc:290] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.358327 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.358955 773 raft_consensus.cc:3058] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.362268 1973 raft_consensus.cc:513] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.362517 848 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.363273 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44769f1170354870ae2d34fdb67ffbae" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.363826 848 leader_election.cc:343] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.363862 2007 raft_consensus.cc:513] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.364943 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfd03ad0bd34b6ca86e7b07a62b2a8e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.365458 848 leader_election.cc:343] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.366362 2007 leader_election.cc:290] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.369570 1973 leader_election.cc:290] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.371408 1672 consensus_queue.cc:1035] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.374799 1672 raft_consensus.cc:491] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.359767 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.361742 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:27.374966 1672 raft_consensus.cc:513] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.375739 1672 leader_election.cc:290] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.376411 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.376986 848 leader_election.cc:343] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.363317 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "44769f1170354870ae2d34fdb67ffbae" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.368227 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cbfd03ad0bd34b6ca86e7b07a62b2a8e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.377846 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.377956 1046 raft_consensus.cc:3058] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.377995 1045 raft_consensus.cc:3058] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.378005 1044 raft_consensus.cc:2466] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.378566 849 leader_election.cc:304] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.378935 1672 raft_consensus.cc:2802] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.379004 1672 raft_consensus.cc:491] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.379065 1672 raft_consensus.cc:3058] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.380837 1046 raft_consensus.cc:2466] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.381206 1048 raft_consensus.cc:3058] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.381495 849 leader_election.cc:304] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.381943 2095 raft_consensus.cc:2802] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.382042 2095 raft_consensus.cc:695] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.382275 2095 consensus_queue.cc:237] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.381108 1045 raft_consensus.cc:2466] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.383801 1048 raft_consensus.cc:2466] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.385532 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab4f69c45295422f85b310aa55220e9b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.388098 984 leader_election.cc:343] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.388566 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ab4f69c45295422f85b310aa55220e9b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.388815 913 raft_consensus.cc:3058] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.391714 913 raft_consensus.cc:2466] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:27.392263 986 leader_election.cc:304] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.395984 849 leader_election.cc:304] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.396600 849 leader_election.cc:304] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.397321 2109 raft_consensus.cc:2802] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.397454 2109 raft_consensus.cc:695] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.397745 2109 consensus_queue.cc:237] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.398737 1997 leader_election.cc:290] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.398823 2108 raft_consensus.cc:2802] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.398936 2108 raft_consensus.cc:695] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.399181 1997 raft_consensus.cc:2802] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.399156 2108 consensus_queue.cc:237] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.399407 1997 raft_consensus.cc:695] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:27.399677 1997 consensus_queue.cc:237] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.400775 2160 raft_consensus.cc:491] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.401049 2160 raft_consensus.cc:513] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.402150 2160 leader_election.cc:290] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.402926 1584 raft_consensus.cc:491] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.403074 1584 raft_consensus.cc:513] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.403721 1584 leader_election.cc:290] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.406055 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c46ea498d6a14d1ebdffea415cfc7114" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.407233 848 leader_election.cc:343] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.414443 711 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:27.420038 1672 raft_consensus.cc:513] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.421633 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.422266 848 leader_election.cc:343] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.422617 1672 leader_election.cc:290] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.426883 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.427065 1045 raft_consensus.cc:3058] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.429713 1045 raft_consensus.cc:2466] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.430611 1584 raft_consensus.cc:491] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.430739 1584 raft_consensus.cc:513] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.431365 1584 leader_election.cc:290] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.432346 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.432898 848 leader_election.cc:343] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.434209 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.434418 1045 raft_consensus.cc:2466] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.436057 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c46ea498d6a14d1ebdffea415cfc7114" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.436615 1048 raft_consensus.cc:2466] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.439481 2108 raft_consensus.cc:491] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.439719 2108 raft_consensus.cc:513] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.440654 2160 raft_consensus.cc:491] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.440908 2160 raft_consensus.cc:513] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.441716 2160 leader_election.cc:290] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.442971 711 leader_election.cc:343] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:27.443272 2108 leader_election.cc:290] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.445637 849 leader_election.cc:304] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.446343 849 leader_election.cc:304] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.448225 2007 raft_consensus.cc:2802] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
W20250905 19:52:27.448923 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 143128us
I20250905 19:52:27.454586 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.456144 2007 raft_consensus.cc:695] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.456573 2007 consensus_queue.cc:237] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.457825 2108 consensus_queue.cc:1035] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.005s
W20250905 19:52:27.458197 848 leader_election.cc:343] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.458328 1672 raft_consensus.cc:2802] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.460712 1672 raft_consensus.cc:491] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.460943 1672 raft_consensus.cc:3058] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.464227 1672 raft_consensus.cc:513] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.476094 1798 raft_consensus.cc:491] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.476419 1798 raft_consensus.cc:513] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.482710 1798 leader_election.cc:290] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.483816 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b0d4be788f841c2860f5dd65275b623" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.484620 984 leader_election.cc:343] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.499194 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 106974us
I20250905 19:52:27.490072 1658 raft_consensus.cc:491] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.505673 1658 raft_consensus.cc:513] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.506418 1658 leader_election.cc:290] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.492250 2108 raft_consensus.cc:491] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.506738 2108 raft_consensus.cc:513] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.511859 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b0d4be788f841c2860f5dd65275b623" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.513286 2108 leader_election.cc:290] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.497067 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 50768.6us
W20250905 19:52:27.499173 848 consensus_peers.cc:489] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.514984 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e15350150db413fa4f100171280f46e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.515817 848 leader_election.cc:343] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.519651 1170 tablet_bootstrap.cc:492] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:27.519837 1170 ts_tablet_manager.cc:1397] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.294s user 0.014s sys 0.010s
I20250905 19:52:27.520686 1170 raft_consensus.cc:357] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.520885 1170 raft_consensus.cc:383] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.520983 1170 raft_consensus.cc:738] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:27.522605 986 leader_election.cc:343] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.523028 986 leader_election.cc:304] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.524056 1798 raft_consensus.cc:2747] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.527588 776 raft_consensus.cc:3058] T 7cdd4de055864d4fa6adb36bf190e609 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.527979 1170 consensus_queue.cc:260] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.531469 1170 ts_tablet_manager.cc:1428] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250905 19:52:27.534853 1170 tablet_bootstrap.cc:492] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.538751 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8b6823580cb4040b49ed0a49b748965" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.539896 1170 tablet_bootstrap.cc:654] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:27.539832 848 leader_election.cc:343] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.540336 849 leader_election.cc:304] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.541361 849 leader_election.cc:304] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.547036 1584 raft_consensus.cc:2802] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.547232 1584 raft_consensus.cc:695] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.547551 1584 consensus_queue.cc:237] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.541708 1658 raft_consensus.cc:2802] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.549749 1658 raft_consensus.cc:491] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.552095 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.552304 1053 raft_consensus.cc:2466] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.554828 1658 raft_consensus.cc:3058] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.555002 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8b6823580cb4040b49ed0a49b748965" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.555174 1050 raft_consensus.cc:2466] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:27.560102 848 consensus_peers.cc:489] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.561453 1672 leader_election.cc:290] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.563300 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.563879 848 leader_election.cc:343] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.565066 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e15350150db413fa4f100171280f46e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.565285 1050 raft_consensus.cc:2466] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.565261 1658 raft_consensus.cc:513] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.566365 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c46ea498d6a14d1ebdffea415cfc7114" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.566910 848 leader_election.cc:343] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.567595 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28e403dc2f6471ebea1805fd843cfb1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.567785 913 raft_consensus.cc:2466] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:27.567214 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.568086 914 raft_consensus.cc:2466] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:27.568238 1658 leader_election.cc:290] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.568832 849 leader_election.cc:304] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.568881 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c46ea498d6a14d1ebdffea415cfc7114" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.569037 1048 raft_consensus.cc:3058] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.569502 849 leader_election.cc:304] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.571161 1672 raft_consensus.cc:2802] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.571283 1672 raft_consensus.cc:491] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.571377 1672 raft_consensus.cc:3058] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.571619 1171 tablet_bootstrap.cc:492] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.571768 1171 ts_tablet_manager.cc:1397] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.270s user 0.046s sys 0.138s
I20250905 19:52:27.572494 712 leader_election.cc:304] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.572651 1171 raft_consensus.cc:357] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.572819 1171 raft_consensus.cc:383] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.572913 1171 raft_consensus.cc:738] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.573143 1171 consensus_queue.cc:260] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.573266 712 leader_election.cc:304] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.573690 1171 ts_tablet_manager.cc:1428] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:27.573762 2100 raft_consensus.cc:2802] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.574014 1171 tablet_bootstrap.cc:492] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.574136 2100 raft_consensus.cc:491] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.574303 2100 raft_consensus.cc:3058] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.575896 1171 tablet_bootstrap.cc:654] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.578006 2154 raft_consensus.cc:2802] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.578125 2154 raft_consensus.cc:491] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.578260 2154 raft_consensus.cc:3058] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.577167 1584 consensus_queue.cc:1035] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:52:27.581818 1584 raft_consensus.cc:2802] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.581959 1584 raft_consensus.cc:491] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.582051 1584 raft_consensus.cc:3058] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.582373 1997 raft_consensus.cc:491] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.582587 1997 raft_consensus.cc:513] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.582639 2154 raft_consensus.cc:513] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.583986 2154 leader_election.cc:290] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.584120 711 leader_election.cc:343] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:27.571933 1048 raft_consensus.cc:2466] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.584481 1997 leader_election.cc:290] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.584620 1584 raft_consensus.cc:513] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.585052 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.585234 1050 raft_consensus.cc:3058] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.585224 1584 leader_election.cc:290] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.585963 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.586473 984 leader_election.cc:343] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.587793 2100 raft_consensus.cc:513] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.588198 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8b6823580cb4040b49ed0a49b748965" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.588517 2100 leader_election.cc:290] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.589087 1050 raft_consensus.cc:2466] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.589694 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:27.590049 848 consensus_peers.cc:489] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:27.590258 848 leader_election.cc:343] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.590370 711 leader_election.cc:343] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1
W20250905 19:52:27.590693 986 leader_election.cc:343] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.590812 986 leader_election.cc:304] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.591041 1798 raft_consensus.cc:2747] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.595628 1658 consensus_queue.cc:1035] T ff2f289c293f41379e4af470d747860c P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.604568 849 leader_election.cc:304] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.604980 1658 raft_consensus.cc:2802] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.605130 1658 raft_consensus.cc:491] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.605307 1658 raft_consensus.cc:3058] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.608157 1658 raft_consensus.cc:513] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.609040 1658 leader_election.cc:290] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.609928 1169 tablet_bootstrap.cc:492] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.610149 1169 ts_tablet_manager.cc:1397] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.365s user 0.008s sys 0.004s
I20250905 19:52:27.610780 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e15350150db413fa4f100171280f46e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.611032 1169 raft_consensus.cc:357] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.611270 1169 raft_consensus.cc:383] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.611368 1169 raft_consensus.cc:738] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:27.611610 848 leader_election.cc:343] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.616322 749 tablet_service.cc:1468] Processing CreateTablet for tablet 31484264112d42b089536d1ecc8ba021 (DEFAULT_TABLE table=default.Table_37507bfcf78648779f20bfe755073f99 [id=cbe19099fb774bdf8204799fb6ce4243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.616657 1022 tablet_service.cc:1468] Processing CreateTablet for tablet 31484264112d42b089536d1ecc8ba021 (DEFAULT_TABLE table=default.Table_37507bfcf78648779f20bfe755073f99 [id=cbe19099fb774bdf8204799fb6ce4243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.617074 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31484264112d42b089536d1ecc8ba021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.617354 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31484264112d42b089536d1ecc8ba021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.625280 849 leader_election.cc:304] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.625957 849 leader_election.cc:304] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.625603 2095 raft_consensus.cc:2802] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.626212 2095 raft_consensus.cc:695] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.626394 2109 raft_consensus.cc:2802] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.626649 2109 raft_consensus.cc:695] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.626986 2109 consensus_queue.cc:237] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.611657 1169 consensus_queue.cc:260] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.628461 1169 ts_tablet_manager.cc:1428] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250905 19:52:27.629763 1997 raft_consensus.cc:491] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.629746 891 tablet_service.cc:1468] Processing CreateTablet for tablet 31484264112d42b089536d1ecc8ba021 (DEFAULT_TABLE table=default.Table_37507bfcf78648779f20bfe755073f99 [id=cbe19099fb774bdf8204799fb6ce4243]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.629921 1997 raft_consensus.cc:513] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.630234 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31484264112d42b089536d1ecc8ba021. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.630584 1997 leader_election.cc:290] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.635156 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.636247 984 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.636641 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.577865 1672 raft_consensus.cc:513] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.637722 754 tablet_service.cc:1468] Processing CreateTablet for tablet ad956757086b4d728698679c63c9b7ee (DEFAULT_TABLE table=default.Table_c1b824c37b3b4413b20c5b2172059490 [id=47bee8087f0943199f82be1345be78fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.638186 1672 leader_election.cc:290] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.638288 754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad956757086b4d728698679c63c9b7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.640127 1169 tablet_bootstrap.cc:492] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:27.640331 986 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.640436 986 leader_election.cc:304] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.641019 1997 raft_consensus.cc:2747] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.641836 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:27.646754 848 leader_election.cc:343] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.647897 1170 tablet_bootstrap.cc:492] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:27.648146 1170 ts_tablet_manager.cc:1397] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.000s
I20250905 19:52:27.649147 1170 raft_consensus.cc:357] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.649405 1170 raft_consensus.cc:383] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.649489 1170 raft_consensus.cc:738] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.654623 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e15350150db413fa4f100171280f46e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.654810 1050 raft_consensus.cc:3058] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.654930 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a8b6823580cb4040b49ed0a49b748965" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.655121 1053 raft_consensus.cc:3058] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.657671 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f28e403dc2f6471ebea1805fd843cfb1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.657727 1053 raft_consensus.cc:2466] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.657866 913 raft_consensus.cc:3058] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.657974 1050 raft_consensus.cc:2466] T 6e15350150db413fa4f100171280f46e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.649780 1170 consensus_queue.cc:260] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.658514 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "78076f6534814939ae8c2db9a4bd1879" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.658650 1170 ts_tablet_manager.cc:1428] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250905 19:52:27.658684 914 raft_consensus.cc:3058] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.658761 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.658922 1046 raft_consensus.cc:3058] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.659672 1169 tablet_bootstrap.cc:654] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.661307 1046 raft_consensus.cc:2466] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.662123 877 tablet_service.cc:1468] Processing CreateTablet for tablet ad956757086b4d728698679c63c9b7ee (DEFAULT_TABLE table=default.Table_c1b824c37b3b4413b20c5b2172059490 [id=47bee8087f0943199f82be1345be78fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.662599 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad956757086b4d728698679c63c9b7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.662806 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 5a5d38136c034b8497045176f749a98d (DEFAULT_TABLE table=default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.663321 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5d38136c034b8497045176f749a98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.663825 1022 tablet_service.cc:1468] Processing CreateTablet for tablet ad956757086b4d728698679c63c9b7ee (DEFAULT_TABLE table=default.Table_c1b824c37b3b4413b20c5b2172059490 [id=47bee8087f0943199f82be1345be78fd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.664183 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ad956757086b4d728698679c63c9b7ee. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.667128 914 raft_consensus.cc:2466] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:27.667434 913 raft_consensus.cc:2466] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:27.668277 2095 consensus_queue.cc:237] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.669414 752 tablet_service.cc:1468] Processing CreateTablet for tablet 5a5d38136c034b8497045176f749a98d (DEFAULT_TABLE table=default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.669613 774 raft_consensus.cc:3058] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.669871 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5d38136c034b8497045176f749a98d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.673942 1170 tablet_bootstrap.cc:492] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.676565 877 tablet_service.cc:1468] Processing CreateTablet for tablet 5a5d38136c034b8497045176f749a98d (DEFAULT_TABLE table=default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.677002 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5a5d38136c034b8497045176f749a98d. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:27.680305 848 consensus_peers.cc:489] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.683727 712 leader_election.cc:304] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.684113 2100 raft_consensus.cc:2802] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.684305 2100 raft_consensus.cc:695] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:27.684666 2100 consensus_queue.cc:237] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.686523 2072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:27.687012 2072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:27.687166 2072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20250905 19:52:27.688165 712 leader_election.cc:304] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.688577 1849 raft_consensus.cc:2802] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.688726 1849 raft_consensus.cc:695] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:27.689013 1849 consensus_queue.cc:237] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.695319 849 leader_election.cc:304] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.697940 1973 raft_consensus.cc:2802] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.698256 1973 raft_consensus.cc:695] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.698558 1973 consensus_queue.cc:237] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.699003 1169 tablet_bootstrap.cc:492] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.699141 1169 ts_tablet_manager.cc:1397] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.060s user 0.000s sys 0.002s
I20250905 19:52:27.699409 849 leader_election.cc:304] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.699878 1584 raft_consensus.cc:2802] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.700060 1584 raft_consensus.cc:695] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.700330 1584 consensus_queue.cc:237] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.701042 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 51d92f92940b4952b68becf86661d459 (DEFAULT_TABLE table=default.Table_5277901ee519486abb3c0e90d83961b5 [id=848f6c88009643beb5a2abcf7d5c35b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.701445 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d92f92940b4952b68becf86661d459. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:27.704003 2072 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20250905 19:52:27.704188 2072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:27.704332 2072 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 19:52:27.704464 2072 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 19:52:27.705507 877 tablet_service.cc:1468] Processing CreateTablet for tablet 51d92f92940b4952b68becf86661d459 (DEFAULT_TABLE table=default.Table_5277901ee519486abb3c0e90d83961b5 [id=848f6c88009643beb5a2abcf7d5c35b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.705935 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d92f92940b4952b68becf86661d459. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.699882 849 leader_election.cc:304] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.708334 2007 raft_consensus.cc:2802] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.709695 1169 raft_consensus.cc:357] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.709885 1169 raft_consensus.cc:383] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.709944 1169 raft_consensus.cc:738] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:27.710381 848 consensus_peers.cc:489] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.710845 752 tablet_service.cc:1468] Processing CreateTablet for tablet 51d92f92940b4952b68becf86661d459 (DEFAULT_TABLE table=default.Table_5277901ee519486abb3c0e90d83961b5 [id=848f6c88009643beb5a2abcf7d5c35b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.711455 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51d92f92940b4952b68becf86661d459. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.712025 2007 raft_consensus.cc:695] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.712384 2007 consensus_queue.cc:237] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.713783 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 2642bb409bb4473db034035b1b834739 (DEFAULT_TABLE table=default.Table_cfe02f65c42d492b9ebd6d9891ff4b34 [id=03ed72cf63184003be5e02061d130ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.714180 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2642bb409bb4473db034035b1b834739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.715548 878 tablet_service.cc:1468] Processing CreateTablet for tablet 2642bb409bb4473db034035b1b834739 (DEFAULT_TABLE table=default.Table_cfe02f65c42d492b9ebd6d9891ff4b34 [id=03ed72cf63184003be5e02061d130ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.716001 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2642bb409bb4473db034035b1b834739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.717964 1171 tablet_bootstrap.cc:492] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.718138 1171 ts_tablet_manager.cc:1397] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.144s user 0.036s sys 0.046s
I20250905 19:52:27.718906 1171 raft_consensus.cc:357] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.719110 1171 raft_consensus.cc:383] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.719190 1171 raft_consensus.cc:738] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.719415 1171 consensus_queue.cc:260] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.719923 1171 ts_tablet_manager.cc:1428] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:27.720258 1171 tablet_bootstrap.cc:492] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:27.721608 2072 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38687
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=46537
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:27.721915 1171 tablet_bootstrap.cc:654] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.723315 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 539f67fac93c46ebb5897ea9f02016fd (DEFAULT_TABLE table=default.Table_af469c2f14f24b01ae62bc25f1e8e4f8 [id=ba5be2a2e05e47e8a32ae828f6c828e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.723719 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539f67fac93c46ebb5897ea9f02016fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.724135 2072 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:27.726697 2072 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:52:27.726679 1016 tablet_service.cc:1468] Processing CreateTablet for tablet c3b2fcc62ff0415f8073edb79b844600 (DEFAULT_TABLE table=default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0 [id=beb46dbcc0634881b6c69db37796ce3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.727061 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b2fcc62ff0415f8073edb79b844600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.730041 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:27.730444 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:27.730887 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:27.731213 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:27.731438 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:27.731597 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:27.731951 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:27.732398 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:27.733536 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:27.733819 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:27.734476 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:27.734834 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:27.735139 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:27.735559 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:27.738276 878 tablet_service.cc:1468] Processing CreateTablet for tablet 539f67fac93c46ebb5897ea9f02016fd (DEFAULT_TABLE table=default.Table_af469c2f14f24b01ae62bc25f1e8e4f8 [id=ba5be2a2e05e47e8a32ae828f6c828e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.738729 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539f67fac93c46ebb5897ea9f02016fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.741093 891 tablet_service.cc:1468] Processing CreateTablet for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (DEFAULT_TABLE table=default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.741127 1028 tablet_service.cc:1468] Processing CreateTablet for tablet cff0e7cc7c814627ad8964fa6a3b1576 (DEFAULT_TABLE table=default.Table_487dad5063df425c8b4a44d7f8adbd39 [id=866aff163c794895abdd7632b46a252b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.741639 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff0e7cc7c814627ad8964fa6a3b1576. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.743541 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9ae8fd5fdc4b539308268ba9729cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.746112 1997 raft_consensus.cc:491] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.746263 1997 raft_consensus.cc:513] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.746891 1997 leader_election.cc:290] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.751500 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.752106 984 leader_election.cc:343] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.747678 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:27.752771 986 leader_election.cc:343] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.752905 986 leader_election.cc:304] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.753300 1997 raft_consensus.cc:2747] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.753237 741 tablet_service.cc:1468] Processing CreateTablet for tablet 539f67fac93c46ebb5897ea9f02016fd (DEFAULT_TABLE table=default.Table_af469c2f14f24b01ae62bc25f1e8e4f8 [id=ba5be2a2e05e47e8a32ae828f6c828e6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.753690 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 539f67fac93c46ebb5897ea9f02016fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.753638 756 tablet_service.cc:1468] Processing CreateTablet for tablet 96aaaf6d94594861b545f831f77dcb55 (DEFAULT_TABLE table=default.Table_6e9009b9124c413eb5058a1f3294990a [id=78d19980c7cc4a79b1b1dccc0a7f674a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.753993 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96aaaf6d94594861b545f831f77dcb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.754308 737 tablet_service.cc:1468] Processing CreateTablet for tablet c3b2fcc62ff0415f8073edb79b844600 (DEFAULT_TABLE table=default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0 [id=beb46dbcc0634881b6c69db37796ce3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.754629 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b2fcc62ff0415f8073edb79b844600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.756647 881 tablet_service.cc:1468] Processing CreateTablet for tablet cff0e7cc7c814627ad8964fa6a3b1576 (DEFAULT_TABLE table=default.Table_487dad5063df425c8b4a44d7f8adbd39 [id=866aff163c794895abdd7632b46a252b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.757495 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 96aaaf6d94594861b545f831f77dcb55 (DEFAULT_TABLE table=default.Table_6e9009b9124c413eb5058a1f3294990a [id=78d19980c7cc4a79b1b1dccc0a7f674a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.758204 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96aaaf6d94594861b545f831f77dcb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.760354 747 tablet_service.cc:1468] Processing CreateTablet for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (DEFAULT_TABLE table=default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.760778 747 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9ae8fd5fdc4b539308268ba9729cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.761010 1170 tablet_bootstrap.cc:654] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.761818 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff0e7cc7c814627ad8964fa6a3b1576. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.762109 1016 tablet_service.cc:1468] Processing CreateTablet for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (DEFAULT_TABLE table=default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.763455 889 tablet_service.cc:1468] Processing CreateTablet for tablet 96aaaf6d94594861b545f831f77dcb55 (DEFAULT_TABLE table=default.Table_6e9009b9124c413eb5058a1f3294990a [id=78d19980c7cc4a79b1b1dccc0a7f674a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.763828 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 96aaaf6d94594861b545f831f77dcb55. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.768364 1849 raft_consensus.cc:491] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.768543 1849 raft_consensus.cc:513] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.768985 749 tablet_service.cc:1468] Processing CreateTablet for tablet 2642bb409bb4473db034035b1b834739 (DEFAULT_TABLE table=default.Table_cfe02f65c42d492b9ebd6d9891ff4b34 [id=03ed72cf63184003be5e02061d130ce3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.769306 1849 leader_election.cc:290] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.769496 1997 raft_consensus.cc:491] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.769614 1997 raft_consensus.cc:513] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.770128 711 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:27.770196 1997 leader_election.cc:290] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.770416 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2642bb409bb4473db034035b1b834739. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.770710 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06850faa25de4b3d83f6bf98ccd370f0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.770933 913 raft_consensus.cc:2466] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:27.770946 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06850faa25de4b3d83f6bf98ccd370f0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.771485 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd9ae8fd5fdc4b539308268ba9729cd6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.771514 986 leader_election.cc:304] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.771854 1997 raft_consensus.cc:2802] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.771944 1997 raft_consensus.cc:491] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.772022 1997 raft_consensus.cc:3058] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:27.773546 984 leader_election.cc:343] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.774571 1997 raft_consensus.cc:513] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.775628 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06850faa25de4b3d83f6bf98ccd370f0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.776170 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06850faa25de4b3d83f6bf98ccd370f0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.776391 913 raft_consensus.cc:3058] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.777442 1672 raft_consensus.cc:491] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.777577 1672 raft_consensus.cc:513] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.777983 889 tablet_service.cc:1468] Processing CreateTablet for tablet 00d1d24868204ce09a7f5e40357fec18 (DEFAULT_TABLE table=default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.778190 1672 leader_election.cc:290] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.778395 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d1d24868204ce09a7f5e40357fec18. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.778754 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:27.779263 913 raft_consensus.cc:2466] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:27.779314 848 leader_election.cc:343] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.779884 984 leader_election.cc:343] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.780185 986 leader_election.cc:304] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.780318 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 00d1d24868204ce09a7f5e40357fec18 (DEFAULT_TABLE table=default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.780536 2163 raft_consensus.cc:2802] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.780682 2163 raft_consensus.cc:695] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:27.781073 756 tablet_service.cc:1468] Processing CreateTablet for tablet 00d1d24868204ce09a7f5e40357fec18 (DEFAULT_TABLE table=default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.781059 2163 consensus_queue.cc:237] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.781453 756 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d1d24868204ce09a7f5e40357fec18. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:27.783479 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 75111.2us
I20250905 19:52:27.784430 1997 leader_election.cc:290] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.786585 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:27.786803 1048 raft_consensus.cc:2466] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:27.780689 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00d1d24868204ce09a7f5e40357fec18. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:27.788592 848 consensus_peers.cc:489] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.791193 1169 consensus_queue.cc:260] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.791420 1798 raft_consensus.cc:491] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.791541 1798 raft_consensus.cc:513] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.791908 1169 ts_tablet_manager.cc:1428] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.093s user 0.000s sys 0.002s
I20250905 19:52:27.792228 1169 tablet_bootstrap.cc:492] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.792254 1798 leader_election.cc:290] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.793645 1169 tablet_bootstrap.cc:654] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.794032 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb72fe6049894527917e1481bcb7b03a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:27.794716 986 leader_election.cc:343] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.796095 748 tablet_service.cc:1468] Processing CreateTablet for tablet cff0e7cc7c814627ad8964fa6a3b1576 (DEFAULT_TABLE table=default.Table_487dad5063df425c8b4a44d7f8adbd39 [id=866aff163c794895abdd7632b46a252b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.796489 748 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cff0e7cc7c814627ad8964fa6a3b1576. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.800014 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.800239 914 raft_consensus.cc:2466] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:27.802085 1672 consensus_queue.cc:1035] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.803825 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb72fe6049894527917e1481bcb7b03a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.804067 848 consensus_peers.cc:489] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:27.804392 984 leader_election.cc:343] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.804564 984 leader_election.cc:304] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.804837 1798 raft_consensus.cc:2747] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.805797 849 leader_election.cc:304] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.807674 1849 raft_consensus.cc:491] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.807842 1849 raft_consensus.cc:513] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.808629 1849 leader_election.cc:290] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.811086 711 leader_election.cc:343] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:27.814617 712 leader_election.cc:304] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.806012 1973 consensus_queue.cc:1035] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.814956 2108 raft_consensus.cc:2802] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.815151 2108 raft_consensus.cc:491] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.815310 2108 raft_consensus.cc:3058] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.815461 2100 raft_consensus.cc:2802] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.815577 2100 raft_consensus.cc:491] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.815676 2100 raft_consensus.cc:3058] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.817335 1849 consensus_queue.cc:1035] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.809845 1171 tablet_bootstrap.cc:492] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:27.817891 1171 ts_tablet_manager.cc:1397] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.098s user 0.003s sys 0.001s
I20250905 19:52:27.823169 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:27.823545 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:27.824501 877 tablet_service.cc:1468] Processing CreateTablet for tablet c3b2fcc62ff0415f8073edb79b844600 (DEFAULT_TABLE table=default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0 [id=beb46dbcc0634881b6c69db37796ce3c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:27.824637 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:27.824828 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:27.825060 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:27.825368 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:27.825781 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:27.826010 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:27.826292 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:27.826615 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:27.829671 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3b2fcc62ff0415f8073edb79b844600. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:27.830103 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6109bdf8f4caf8c223a42401b94ab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:27.830299 913 raft_consensus.cc:2466] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:27.836747 2100 raft_consensus.cc:513] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.837606 1517 catalog_manager.cc:5582] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:27.838500 1973 consensus_queue.cc:1035] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.839387 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:27.840026 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:27.840320 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:27.840705 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:27.840970 1517 catalog_manager.cc:5582] T ac2e51a9071c4e1dae9e147687291175 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:27.841671 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_9da2eedb50e64356ba6a49d93bafbe28 [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 1 got 0
I20250905 19:52:27.841931 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_849b4e364b3f4ff2b47a2347f84d6b0d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 1 got 0
I20250905 19:52:27.842406 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
W20250905 19:52:27.843108 1542 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet e63b27ff4e434d76a8280f90c22f4082 consensus state has no leader
I20250905 19:52:27.843318 1542 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 31) took 1751 ms. Trace:
I20250905 19:52:27.844130 2108 raft_consensus.cc:513] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.844820 2108 leader_election.cc:290] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.843438 1542 rpcz_store.cc:276] 0905 19:52:26.091459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.463680 (+372221us) service_pool.cc:225] Handling call
0905 19:52:26.464014 (+ 334us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.464240 (+ 226us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:26.464452 (+ 212us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:26.464534 (+ 82us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:26.513936 (+ 49402us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:27.843286 (+1329350us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.472858 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.472983 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.472988 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:26.472990 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:26.473098 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.473108 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.473112 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:26.473166 (+ 54us) tablet.cc:699] Row locks acquired
0905 19:52:26.473169 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:26.473201 (+ 32us) write_op.cc:270] Start()
0905 19:52:26.508683 (+ 35482us) write_op.cc:276] Timestamp: P: 1757101946473185 usec, L: 0
0905 19:52:26.508691 (+ 8us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.509675 (+ 984us) log.cc:844] Serialized 405 byte log entry
0905 19:52:26.513258 (+ 3583us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.513469 (+ 211us) write_op.cc:301] APPLY: starting
0905 19:52:26.513487 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.513529 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.513564 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.513614 (+ 50us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.513647 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:26.513655 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:26.513749 (+ 94us) log.cc:844] Serialized 23 byte log entry
0905 19:52:26.513813 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.513835 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:26.513842 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:26.515056 (+ 1214us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.515062 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":154,"apply.run_cpu_time_us":445,"apply.run_wall_time_us":445,"mutex_wait_us":874,"num_ops":1,"prepare.queue_time_us":7843,"prepare.run_cpu_time_us":991,"prepare.run_wall_time_us":42188,"raft.queue_time_us":1970,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":287,"replication_time_us":4535}]]}
I20250905 19:52:27.856225 1546 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 37894e39e6f2417fbfcd6d43f1ea2b37
I20250905 19:52:27.857098 1546 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 30) took 1765 ms. Trace:
I20250905 19:52:27.857245 1546 rpcz_store.cc:276] 0905 19:52:26.091371 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.459890 (+368519us) service_pool.cc:225] Handling call
0905 19:52:26.462766 (+ 2876us) spinlock_profiling.cc:243] Waited 2.36 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec7ea6 00007fabe1d00466 00007fabe1d00195 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:26.463383 (+ 617us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.463528 (+ 145us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:26.463634 (+ 106us) catalog_manager.cc:2799] Locking tablets
0905 19:52:26.483386 (+ 19752us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:26.523938 (+ 40552us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:26.524109 (+ 171us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:27.856135 (+1332026us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:27.857013 (+ 878us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.515162 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.515268 (+ 106us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.515274 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:26.515277 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:26.515390 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.515404 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.515408 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:26.515521 (+ 113us) tablet.cc:699] Row locks acquired
0905 19:52:26.515524 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:26.515559 (+ 35us) write_op.cc:270] Start()
0905 19:52:26.515596 (+ 37us) write_op.cc:276] Timestamp: P: 1757101946515543 usec, L: 0
0905 19:52:26.515600 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.517541 (+ 1941us) log.cc:844] Serialized 725 byte log entry
0905 19:52:26.523124 (+ 5583us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.523259 (+ 135us) write_op.cc:301] APPLY: starting
0905 19:52:26.523274 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.523320 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.523352 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.523425 (+ 73us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.523459 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:26.523467 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:26.523541 (+ 74us) log.cc:844] Serialized 29 byte log entry
0905 19:52:26.523597 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.523617 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:26.523624 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:26.523687 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.523691 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":5429120,"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":421,"apply.run_wall_time_us":421,"num_ops":2,"prepare.queue_time_us":30590,"prepare.run_cpu_time_us":737,"prepare.run_wall_time_us":2823,"raft.queue_time_us":30,"raft.run_cpu_time_us":3,"raft.run_wall_time_us":4,"replication_time_us":7490}]]}
I20250905 19:52:27.862269 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:27.863111 848 consensus_peers.cc:489] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.865638 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:27.865826 1045 raft_consensus.cc:3058] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.868698 1045 raft_consensus.cc:2466] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:27.868924 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:27.870565 1169 tablet_bootstrap.cc:492] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:27.870699 1169 ts_tablet_manager.cc:1397] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.002s
W20250905 19:52:27.875052 1540 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet dc95a83ca872495894ef439cdda4e364 consensus state has no leader
I20250905 19:52:27.875932 849 leader_election.cc:304] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:27.877768 2095 raft_consensus.cc:2802] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.878710 2095 raft_consensus.cc:695] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:27.879175 2095 consensus_queue.cc:237] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:27.880700 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e63b27ff4e434d76a8280f90c22f4082 consensus state has no leader
I20250905 19:52:27.887410 2163 raft_consensus.cc:491] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.887576 2163 raft_consensus.cc:513] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.888801 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efceaf3c7bda44ddabd5d38130af3dd2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:27.890226 986 leader_election.cc:343] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.890491 2163 leader_election.cc:290] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.897964 1536 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 813f208881ac4782874138987a710d93
I20250905 19:52:27.898800 1536 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 28) took 1807 ms. Trace:
I20250905 19:52:27.898949 1536 rpcz_store.cc:276] 0905 19:52:26.091164 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.332745 (+241581us) service_pool.cc:225] Handling call
0905 19:52:26.361411 (+ 28666us) spinlock_profiling.cc:243] Waited 28.2 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec7ea6 00007fabe1d00466 00007fabe1d00195 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:26.361815 (+ 404us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.362043 (+ 228us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:26.362123 (+ 80us) catalog_manager.cc:2799] Locking tablets
0905 19:52:26.362272 (+ 149us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:26.468695 (+106423us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:26.471327 (+ 2632us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:27.897851 (+1426524us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:27.898748 (+ 897us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.448372 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.456449 (+ 8077us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.456457 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:26.456459 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:26.457310 (+ 851us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.457320 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.457325 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:26.458121 (+ 796us) tablet.cc:699] Row locks acquired
0905 19:52:26.458124 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:26.458332 (+ 208us) write_op.cc:270] Start()
0905 19:52:26.458524 (+ 192us) write_op.cc:276] Timestamp: P: 1757101946458307 usec, L: 0
0905 19:52:26.458528 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.462346 (+ 3818us) log.cc:844] Serialized 759 byte log entry
0905 19:52:26.468054 (+ 5708us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.468162 (+ 108us) write_op.cc:301] APPLY: starting
0905 19:52:26.468178 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.468222 (+ 44us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.468252 (+ 30us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.468333 (+ 81us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.468364 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:26.468371 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:26.468436 (+ 65us) log.cc:844] Serialized 29 byte log entry
0905 19:52:26.468498 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.468517 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:26.468524 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:26.468590 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.468594 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":64918912,"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":425,"apply.run_wall_time_us":425,"num_ops":2,"prepare.queue_time_us":85494,"prepare.run_cpu_time_us":1083,"prepare.run_wall_time_us":18207,"raft.queue_time_us":34,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":9491}]]}
I20250905 19:52:27.905974 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 813f208881ac4782874138987a710d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:27.908622 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 813f208881ac4782874138987a710d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:27.913465 1169 raft_consensus.cc:357] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.913676 1169 raft_consensus.cc:383] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.913733 1169 raft_consensus.cc:738] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.914958 1169 consensus_queue.cc:260] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.838400 712 leader_election.cc:304] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
W20250905 19:52:27.925441 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 87094.6us
I20250905 19:52:27.926146 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:27.926343 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 813f208881ac4782874138987a710d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:27.926419 1849 raft_consensus.cc:2802] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:27.926550 1849 raft_consensus.cc:491] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:27.926679 1849 raft_consensus.cc:3058] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.930598 1849 raft_consensus.cc:513] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.935389 1849 leader_election.cc:290] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:27.839020 1170 tablet_bootstrap.cc:492] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:27.935747 1170 ts_tablet_manager.cc:1397] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.262s user 0.002s sys 0.002s
I20250905 19:52:27.937170 1170 raft_consensus.cc:357] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.937381 1170 raft_consensus.cc:383] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:27.937479 1170 raft_consensus.cc:738] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:27.937727 1170 consensus_queue.cc:260] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.938300 1170 ts_tablet_manager.cc:1428] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:27.938700 1547 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_c30c09bc294d465da486648091481ed1 [id=3118b283e4194c5fa5c5a7e270d14e49] alter complete (version 2)
W20250905 19:52:27.940797 711 leader_election.cc:343] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:27.941324 1170 tablet_bootstrap.cc:492] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:27.943629 1170 tablet_bootstrap.cc:654] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.858925 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:27.889720 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efceaf3c7bda44ddabd5d38130af3dd2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.945271 984 leader_election.cc:343] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.945405 984 leader_election.cc:304] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:27.946118 848 leader_election.cc:343] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.949666 2100 leader_election.cc:290] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:27.957046 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37894e39e6f2417fbfcd6d43f1ea2b37 already in progress: creating tablet
I20250905 19:52:27.957512 2216 tablet_replica.cc:331] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:27.957749 2216 raft_consensus.cc:2241] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:27.957999 2216 raft_consensus.cc:2270] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:27.958148 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8ed6109bdf8f4caf8c223a42401b94ab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.958302 913 raft_consensus.cc:3058] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.958832 2216 ts_tablet_manager.cc:1905] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:27.961117 913 raft_consensus.cc:2466] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:27.966267 2216 ts_tablet_manager.cc:1918] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:27.966410 2216 log.cc:1199] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/813f208881ac4782874138987a710d93
I20250905 19:52:27.966753 1798 raft_consensus.cc:491] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:27.966815 2216 ts_tablet_manager.cc:1939] T 813f208881ac4782874138987a710d93 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:27.966905 1798 raft_consensus.cc:513] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.967465 1798 leader_election.cc:290] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:27.968043 1997 raft_consensus.cc:2747] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.968976 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:27.972571 984 leader_election.cc:343] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:27.973896 711 leader_election.cc:343] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:27.976140 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:27.976651 986 leader_election.cc:343] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:27.976749 986 leader_election.cc:304] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:27.977164 1798 raft_consensus.cc:2747] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:27.979458 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 813f208881ac4782874138987a710d93 (table default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]) successfully deleted
W20250905 19:52:27.981544 711 consensus_peers.cc:489] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.986047 1169 ts_tablet_manager.cc:1428] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.115s user 0.000s sys 0.002s
W20250905 19:52:27.987483 848 consensus_peers.cc:489] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:27.991432 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0267ac3e3f1a45978807737eca5d91ef" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:27.992126 712 leader_election.cc:304] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.992220 913 raft_consensus.cc:3058] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:27.992556 2160 raft_consensus.cc:2802] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.992776 2160 raft_consensus.cc:695] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:27.993166 2160 consensus_queue.cc:237] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:27.994405 1169 tablet_bootstrap.cc:492] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:27.995219 913 raft_consensus.cc:2466] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:27.995718 712 leader_election.cc:304] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:27.995932 1584 consensus_queue.cc:1035] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:27.996048 2100 raft_consensus.cc:2802] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:27.996191 2100 raft_consensus.cc:695] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:27.996330 1169 tablet_bootstrap.cc:654] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:27.996847 2100 consensus_queue.cc:237] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.013053 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37894e39e6f2417fbfcd6d43f1ea2b37 already in progress: creating tablet
I20250905 19:52:28.013132 1170 tablet_bootstrap.cc:492] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.013768 2206 tablet_replica.cc:331] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:28.014605 2206 raft_consensus.cc:2241] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:28.015020 1170 ts_tablet_manager.cc:1397] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.075s user 0.002s sys 0.003s
I20250905 19:52:28.015447 2206 raft_consensus.cc:2270] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:28.017302 1170 raft_consensus.cc:357] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.017477 2206 ts_tablet_manager.cc:1905] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:28.017668 1170 raft_consensus.cc:383] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.017755 1170 raft_consensus.cc:738] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.018187 2163 raft_consensus.cc:491] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.018322 2163 raft_consensus.cc:513] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.018345 1170 consensus_queue.cc:260] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.018958 2163 leader_election.cc:290] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.019899 1170 ts_tablet_manager.cc:1428] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:28.020325 1170 tablet_bootstrap.cc:492] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.020512 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.021353 986 leader_election.cc:343] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.021936 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:28.022454 984 leader_election.cc:343] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.022574 984 leader_election.cc:304] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.022969 1170 tablet_bootstrap.cc:654] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.023037 2163 raft_consensus.cc:2747] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.028453 1169 tablet_bootstrap.cc:492] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.028621 1169 ts_tablet_manager.cc:1397] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.001s
I20250905 19:52:28.029628 1169 raft_consensus.cc:357] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.029790 1169 raft_consensus.cc:383] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.029734 2227 tablet.cc:1722] T 415508e72ee34bc391c4369b5f8a2f48 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.029836 1169 raft_consensus.cc:738] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.030974 1169 consensus_queue.cc:260] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.031559 1169 ts_tablet_manager.cc:1428] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:28.031883 1169 tablet_bootstrap.cc:492] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.034179 1169 tablet_bootstrap.cc:654] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.034582 2108 consensus_queue.cc:1035] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.036775 2109 raft_consensus.cc:491] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.036996 2109 raft_consensus.cc:513] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.037762 2109 leader_election.cc:290] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.049697 1170 tablet_bootstrap.cc:492] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.050626 1170 ts_tablet_manager.cc:1397] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.030s user 0.000s sys 0.013s
I20250905 19:52:28.052184 1170 raft_consensus.cc:357] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.054131 1170 raft_consensus.cc:383] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.054507 1170 raft_consensus.cc:738] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.055258 1170 consensus_queue.cc:260] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.056890 1170 ts_tablet_manager.cc:1428] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:28.039426 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.058140 1053 raft_consensus.cc:2466] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:28.059181 849 leader_election.cc:304] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:28.060722 2109 raft_consensus.cc:2802] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.060834 2109 raft_consensus.cc:491] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.060932 2109 raft_consensus.cc:3058] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.068205 2227 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.069509 2109 raft_consensus.cc:513] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.036288 2228 tablet.cc:1722] T 415508e72ee34bc391c4369b5f8a2f48 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.072389 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:28.074292 1053 raft_consensus.cc:3058] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.078225 1540 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 26) took 1987 ms. Trace:
I20250905 19:52:28.078583 1540 rpcz_store.cc:276] 0905 19:52:26.090996 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.312035 (+221039us) service_pool.cc:225] Handling call
0905 19:52:26.330089 (+ 18054us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.330370 (+ 281us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:26.330575 (+ 205us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:26.330646 (+ 71us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:26.456973 (+126327us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:28.078169 (+1621196us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.392380 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.393172 (+ 792us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.393182 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:26.393187 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:26.393794 (+ 607us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.393962 (+ 168us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.393971 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:26.394455 (+ 484us) tablet.cc:699] Row locks acquired
0905 19:52:26.394461 (+ 6us) write_op.cc:260] PREPARE: finished
0905 19:52:26.424195 (+ 29734us) write_op.cc:270] Start()
0905 19:52:26.424469 (+ 274us) write_op.cc:276] Timestamp: P: 1757101946424020 usec, L: 0
0905 19:52:26.424474 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.427284 (+ 2810us) log.cc:844] Serialized 405 byte log entry
0905 19:52:26.455586 (+ 28302us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.456478 (+ 892us) write_op.cc:301] APPLY: starting
0905 19:52:26.456488 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.456515 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.456536 (+ 21us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.456572 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.456617 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:26.456624 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:26.456674 (+ 50us) log.cc:844] Serialized 23 byte log entry
0905 19:52:26.456711 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.456721 (+ 10us) write_op.cc:454] Released schema lock
0905 19:52:26.456726 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:26.456787 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.456793 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":860,"apply.run_cpu_time_us":300,"apply.run_wall_time_us":300,"num_ops":1,"prepare.queue_time_us":61056,"prepare.run_cpu_time_us":1227,"prepare.run_wall_time_us":36212,"raft.queue_time_us":319,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":7,"replication_time_us":31099}]]}
I20250905 19:52:28.080178 2109 leader_election.cc:290] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.080737 1540 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_9c5296ef0f424c5591eacb019ed29c7d" } modify_external_catalogs: true
I20250905 19:52:28.081216 1540 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_9c5296ef0f424c5591eacb019ed29c7d" } modify_external_catalogs: true
I20250905 19:52:28.081841 1522 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 24) took 1991 ms. Trace:
I20250905 19:52:28.082039 1522 rpcz_store.cc:276] 0905 19:52:26.090815 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.304507 (+213692us) service_pool.cc:225] Handling call
0905 19:52:26.309593 (+ 5086us) spinlock_profiling.cc:243] Waited 4.83 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec9146 00007fabe1d02126 00007fabe1d01e55 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:26.309854 (+ 261us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.310072 (+ 218us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:26.326137 (+ 16065us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:26.326310 (+ 173us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:26.456871 (+130561us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:28.081800 (+1624929us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.331225 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.331355 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.331364 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:26.331369 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:26.372663 (+ 41294us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.372684 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.372694 (+ 10us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:26.373258 (+ 564us) tablet.cc:699] Row locks acquired
0905 19:52:26.373262 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:26.373468 (+ 206us) write_op.cc:270] Start()
0905 19:52:26.373676 (+ 208us) write_op.cc:276] Timestamp: P: 1757101946373444 usec, L: 0
0905 19:52:26.373835 (+ 159us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.375380 (+ 1545us) log.cc:844] Serialized 405 byte log entry
0905 19:52:26.455396 (+ 80016us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.455514 (+ 118us) write_op.cc:301] APPLY: starting
0905 19:52:26.455531 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.455583 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.455670 (+ 87us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.455775 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.455808 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:26.455815 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:26.455897 (+ 82us) log.cc:844] Serialized 23 byte log entry
0905 19:52:26.456337 (+ 440us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.456349 (+ 12us) write_op.cc:454] Released schema lock
0905 19:52:26.456353 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:26.456412 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.456419 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":11118976,"child_traces":[["op",{"apply.queue_time_us":34,"apply.run_cpu_time_us":600,"apply.run_wall_time_us":897,"num_ops":1,"prepare.queue_time_us":4292,"prepare.run_cpu_time_us":1164,"prepare.run_wall_time_us":45294,"raft.queue_time_us":204,"raft.run_cpu_time_us":17,"raft.run_wall_time_us":19,"replication_time_us":81527}]]}
I20250905 19:52:28.084431 1522 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_d66d859b633f453393a574fa8885329b" } modify_external_catalogs: true
I20250905 19:52:28.084708 1522 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_d66d859b633f453393a574fa8885329b" } modify_external_catalogs: true
I20250905 19:52:28.085309 1539 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 27) took 1994 ms. Trace:
I20250905 19:52:28.085242 2230 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.085820 1539 rpcz_store.cc:276] 0905 19:52:26.091092 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.313168 (+222076us) service_pool.cc:225] Handling call
0905 19:52:26.328834 (+ 15666us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.329170 (+ 336us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:26.329388 (+ 218us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:26.329469 (+ 81us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:26.456729 (+127260us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:28.085270 (+1628541us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.386842 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.387605 (+ 763us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.387612 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:26.387615 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:26.388392 (+ 777us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.388410 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.388420 (+ 10us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:26.388983 (+ 563us) tablet.cc:699] Row locks acquired
0905 19:52:26.388987 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:26.389187 (+ 200us) write_op.cc:270] Start()
0905 19:52:26.389465 (+ 278us) write_op.cc:276] Timestamp: P: 1757101946389165 usec, L: 0
0905 19:52:26.389472 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.390837 (+ 1365us) log.cc:844] Serialized 405 byte log entry
0905 19:52:26.455544 (+ 64707us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.456198 (+ 654us) write_op.cc:301] APPLY: starting
0905 19:52:26.456210 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.456239 (+ 29us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.456267 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.456311 (+ 44us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.456340 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:26.456346 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:26.456404 (+ 58us) log.cc:844] Serialized 23 byte log entry
0905 19:52:26.456447 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.456456 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:26.456461 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:26.456509 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.456512 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":621,"apply.run_cpu_time_us":309,"apply.run_wall_time_us":308,"num_ops":1,"prepare.queue_time_us":55074,"prepare.run_cpu_time_us":1220,"prepare.run_wall_time_us":6917,"raft.queue_time_us":153,"raft.run_cpu_time_us":7,"raft.run_wall_time_us":8,"replication_time_us":66053}]]}
I20250905 19:52:28.087752 1539 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_9067a9bea2cc48ecab23b9fd1749474c" } modify_external_catalogs: true
I20250905 19:52:28.088045 1539 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_9067a9bea2cc48ecab23b9fd1749474c" } modify_external_catalogs: true
I20250905 19:52:28.059607 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.088527 1543 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 22) took 1998 ms. Trace:
I20250905 19:52:28.089078 1170 tablet_bootstrap.cc:492] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.089025 1543 rpcz_store.cc:276] 0905 19:52:26.090439 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.115842 (+ 25403us) service_pool.cc:225] Handling call
0905 19:52:26.228380 (+112538us) spinlock_profiling.cc:243] Waited 110 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec9146 00007fabe1d02126 00007fabe1d01e55 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:26.228846 (+ 466us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:26.229368 (+ 522us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:26.229671 (+ 303us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:26.229861 (+ 190us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:26.333301 (+103440us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:28.088476 (+1755175us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:26.314293 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:26.314387 (+ 94us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:26.314392 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:26.314394 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:26.314474 (+ 80us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:26.314482 (+ 8us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:26.314486 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:26.314535 (+ 49us) tablet.cc:699] Row locks acquired
0905 19:52:26.314537 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:26.314567 (+ 30us) write_op.cc:270] Start()
0905 19:52:26.314593 (+ 26us) write_op.cc:276] Timestamp: P: 1757101946314552 usec, L: 0
0905 19:52:26.314597 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:26.314879 (+ 282us) log.cc:844] Serialized 405 byte log entry
0905 19:52:26.331530 (+ 16651us) op_driver.cc:464] REPLICATION: finished
0905 19:52:26.332899 (+ 1369us) write_op.cc:301] APPLY: starting
0905 19:52:26.332908 (+ 9us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:26.332935 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:26.332979 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:26.333018 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:26.333066 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:26.333073 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:26.333127 (+ 54us) log.cc:844] Serialized 23 byte log entry
0905 19:52:26.333168 (+ 41us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:26.333179 (+ 11us) write_op.cc:454] Released schema lock
0905 19:52:26.333184 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":252158208,"child_traces":[["op",{"apply.queue_time_us":1334,"apply.run_cpu_time_us":341,"apply.run_wall_time_us":341,"num_ops":1,"prepare.queue_time_us":73732,"prepare.run_cpu_time_us":565,"prepare.run_wall_time_us":1050,"raft.queue_time_us":40,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":16916,"thread_start_us":27270,"threads_started":1}]]}
I20250905 19:52:28.090721 1543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_24293fcc0f3244c4a016cbb7e189a24d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:28.091593 1543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_24293fcc0f3244c4a016cbb7e189a24d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:28.092089 1543 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 35) took 2000 ms. Trace:
I20250905 19:52:28.092269 1543 rpcz_store.cc:276] 0905 19:52:26.091857 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:28.090498 (+1998641us) service_pool.cc:225] Handling call
0905 19:52:28.091817 (+ 1319us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:28.092002 (+ 185us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:28.093477 1170 tablet_bootstrap.cc:654] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.093545 1543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_3a4d5c41d6f44c128dbb1ef09d643d84"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:28.100642 1053 raft_consensus.cc:2466] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:28.101415 849 leader_election.cc:304] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:28.102201 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:28.102452 2109 raft_consensus.cc:2802] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.102607 2109 raft_consensus.cc:695] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:28.102922 2109 consensus_queue.cc:237] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.104640 848 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.112942 1169 tablet_bootstrap.cc:492] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.116230 1169 ts_tablet_manager.cc:1397] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.010s
I20250905 19:52:28.117931 1169 raft_consensus.cc:357] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.120934 1170 tablet_bootstrap.cc:492] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.122422 1170 ts_tablet_manager.cc:1397] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.006s
I20250905 19:52:28.122666 1169 raft_consensus.cc:383] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.123000 1169 raft_consensus.cc:738] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.124187 1169 consensus_queue.cc:260] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.128751 1170 raft_consensus.cc:357] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.129304 1169 ts_tablet_manager.cc:1428] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250905 19:52:28.129837 1169 tablet_bootstrap.cc:492] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.131691 1170 raft_consensus.cc:383] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.131808 1170 raft_consensus.cc:738] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.132105 1170 consensus_queue.cc:260] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.132900 1169 tablet_bootstrap.cc:654] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:28.133178 1543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:28.129082 2235 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.129137 2160 consensus_queue.cc:1035] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.132712 1170 ts_tablet_manager.cc:1428] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250905 19:52:28.146615 1170 tablet_bootstrap.cc:492] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:28.147759 2240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:28.149582 711 consensus_peers.cc:489] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.152346 1169 tablet_bootstrap.cc:492] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.152501 1169 ts_tablet_manager.cc:1397] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.007s
I20250905 19:52:28.153072 1170 tablet_bootstrap.cc:654] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.153308 1169 raft_consensus.cc:357] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.153491 1169 raft_consensus.cc:383] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.153548 1169 raft_consensus.cc:738] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.153978 1169 consensus_queue.cc:260] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.154520 1169 ts_tablet_manager.cc:1428] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:28.154811 2230 tablet.cc:1722] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.154807 1169 tablet_bootstrap.cc:492] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.158363 1169 tablet_bootstrap.cc:654] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:28.158705 2244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:28.159933 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 55312.1us
W20250905 19:52:28.160245 848 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.161069 2072 server_base.cc:1047] running on GCE node
I20250905 19:52:28.164098 2227 tablet.cc:1722] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.177855 2007 consensus_queue.cc:1035] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.181455 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:28.188599 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37894e39e6f2417fbfcd6d43f1ea2b37 already in progress: creating tablet
I20250905 19:52:28.189194 2205 tablet_replica.cc:331] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:28.189631 2205 raft_consensus.cc:2241] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:28.190093 2205 raft_consensus.cc:2270] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:28.190872 2160 raft_consensus.cc:491] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.191105 2160 raft_consensus.cc:513] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.191736 984 consensus_peers.cc:489] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.191913 2160 leader_election.cc:290] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.192842 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d8144bb4f14358af539b7aae326918" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.193002 2205 ts_tablet_manager.cc:1905] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:28.193096 913 raft_consensus.cc:2466] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:28.194469 712 leader_election.cc:304] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:28.195298 1584 raft_consensus.cc:491] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.195456 1584 raft_consensus.cc:513] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.196902 1584 leader_election.cc:290] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.197798 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.198019 1047 raft_consensus.cc:2466] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:28.198076 2160 raft_consensus.cc:2802] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.198192 2160 raft_consensus.cc:491] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20250905 19:52:28.198298 711 leader_election.cc:343] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918
I20250905 19:52:28.198526 849 leader_election.cc:304] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:28.198302 2160 raft_consensus.cc:3058] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.198985 1584 raft_consensus.cc:2802] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.199059 1584 raft_consensus.cc:491] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.199149 1584 raft_consensus.cc:3058] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.201830 1584 raft_consensus.cc:513] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.202736 2160 raft_consensus.cc:513] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.203157 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:28.203333 1047 raft_consensus.cc:3058] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.203948 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d8144bb4f14358af539b7aae326918" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:28.204128 913 raft_consensus.cc:3058] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:28.204792 711 leader_election.cc:343] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918
I20250905 19:52:28.205011 2160 leader_election.cc:290] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.206034 1047 raft_consensus.cc:2466] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:28.206490 849 leader_election.cc:304] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:28.206562 913 raft_consensus.cc:2466] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:28.206867 2108 raft_consensus.cc:2802] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.206985 2108 raft_consensus.cc:695] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:28.206992 712 leader_election.cc:304] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.207227 2108 consensus_queue.cc:237] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.209437 1584 leader_election.cc:290] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.213068 2160 raft_consensus.cc:2802] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.212905 1584 consensus_queue.cc:1035] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.216014 2160 raft_consensus.cc:695] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:28.216516 2160 consensus_queue.cc:237] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.218039 2235 tablet.cc:1722] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
W20250905 19:52:28.220347 2204 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:28.221740 2072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:28.237242 2072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:28.250733 1170 tablet_bootstrap.cc:492] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.250926 1170 ts_tablet_manager.cc:1397] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.104s user 0.014s sys 0.029s
I20250905 19:52:28.252025 1170 raft_consensus.cc:357] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.252287 1170 raft_consensus.cc:383] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.252375 1170 raft_consensus.cc:738] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.252702 1170 consensus_queue.cc:260] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.253443 1170 ts_tablet_manager.cc:1428] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250905 19:52:28.254484 2072 hybrid_clock.cc:648] HybridClock initialized: now 1757101948254350 us; error 129 us; skew 500 ppm
W20250905 19:52:28.255039 848 consensus_peers.cc:489] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.255069 2072 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20250905 19:52:28.255249 848 consensus_peers.cc:489] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:28.257369 711 consensus_peers.cc:489] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.257642 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.258271 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:28.265785 1472 consensus_peers.cc:489] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e -> Peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Couldn't send request to peer c401c9f3020b4ae5931c566c6bcb1711. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250905 19:52:28.265758 1798 consensus_queue.cc:1035] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:52:28.269203 1170 tablet_bootstrap.cc:492] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.273092 1543 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 36) took 2181 ms. Trace:
I20250905 19:52:28.273319 1543 rpcz_store.cc:276] 0905 19:52:26.091946 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:28.093284 (+2001338us) service_pool.cc:225] Handling call
0905 19:52:28.272806 (+179522us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:28.273012 (+ 206us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:28.276460 1170 tablet_bootstrap.cc:654] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:28.277675 848 consensus_peers.cc:489] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.277959 1849 raft_consensus.cc:491] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.278131 1849 raft_consensus.cc:513] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.278189 1171 raft_consensus.cc:357] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.278333 1171 raft_consensus.cc:383] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.278378 1171 raft_consensus.cc:738] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:28.278461 848 consensus_peers.cc:489] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:28.278625 848 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.278828 1849 leader_election.cc:290] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.278934 848 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.279459 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.278550 1171 consensus_queue.cc:260] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.281816 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "61667a379974487195f81043683b9a24" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.282459 711 leader_election.cc:304] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.282586 774 raft_consensus.cc:3058] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.284446 1171 ts_tablet_manager.cc:1428] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.466s user 0.017s sys 0.000s
I20250905 19:52:28.284799 1171 tablet_bootstrap.cc:492] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.286245 1171 tablet_bootstrap.cc:654] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.286751 2072 webserver.cc:480] Webserver started at http://127.0.104.62:46537/ using document root <none> and password file <none>
W20250905 19:52:28.287191 848 consensus_peers.cc:489] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.288049 2072 fs_manager.cc:362] Metadata directory not provided
W20250905 19:52:28.288196 848 consensus_peers.cc:489] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.288223 2072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:28.290990 2100 raft_consensus.cc:2747] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.292369 1171 tablet_bootstrap.cc:492] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.293283 1170 tablet_bootstrap.cc:492] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.293565 1170 ts_tablet_manager.cc:1397] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.001s
I20250905 19:52:28.294656 1170 raft_consensus.cc:357] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.295042 1170 raft_consensus.cc:383] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.295182 1170 raft_consensus.cc:738] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.295565 1170 consensus_queue.cc:260] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.297622 1170 ts_tablet_manager.cc:1428] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:28.299124 1170 tablet_bootstrap.cc:492] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.299361 1171 ts_tablet_manager.cc:1397] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.000s
I20250905 19:52:28.301705 1170 tablet_bootstrap.cc:654] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.302665 1171 raft_consensus.cc:357] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.302961 1171 raft_consensus.cc:383] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.303071 1171 raft_consensus.cc:738] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.303458 1171 consensus_queue.cc:260] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.304184 1171 ts_tablet_manager.cc:1428] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:28.305497 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:28.305965 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:28.306555 1171 tablet_bootstrap.cc:492] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.306560 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b2c7906a718c46d9a76587a70d96a1f9 (table default.Table_a82a3756e48a4bfc99f4aacf9b57f6d3 [id=99384aac4e624d6db630b249c3ba2543]). Expected version 2 got 1
I20250905 19:52:28.306970 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:28.308473 1169 tablet_bootstrap.cc:492] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.308658 1169 ts_tablet_manager.cc:1397] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.154s user 0.040s sys 0.040s
I20250905 19:52:28.308729 1171 tablet_bootstrap.cc:654] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.309588 1169 raft_consensus.cc:357] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.309865 1169 raft_consensus.cc:383] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.310041 1169 raft_consensus.cc:738] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.310338 1169 consensus_queue.cc:260] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.311007 1169 ts_tablet_manager.cc:1428] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:28.311504 1169 tablet_bootstrap.cc:492] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.313421 1169 tablet_bootstrap.cc:654] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.314357 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:28.314949 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:28.316119 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 2 got 1
I20250905 19:52:28.317108 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:28.317852 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:28.318653 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:28.319576 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:28.320163 1798 raft_consensus.cc:491] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.320320 1798 raft_consensus.cc:513] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.321061 1798 leader_election.cc:290] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.321122 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:28.322485 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:28.322774 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.324008 986 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.324127 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:28.324345 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:28.324656 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.325731 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:28.326383 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:28.327227 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
W20250905 19:52:28.327510 984 consensus_peers.cc:489] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:28.327708 984 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.327823 984 leader_election.cc:304] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.328267 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:28.328452 2163 raft_consensus.cc:2747] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.328624 2163 raft_consensus.cc:491] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.328734 2163 raft_consensus.cc:513] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.329267 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:28.329414 2163 leader_election.cc:290] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.329538 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:28.330293 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3315d3e9633e42ea8d7a6b5934cb0438" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.330892 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
W20250905 19:52:28.330972 986 leader_election.cc:343] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.331733 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3315d3e9633e42ea8d7a6b5934cb0438" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.331851 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:28.332049 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:28.332212 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:28.333249 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:28.333892 1171 tablet_bootstrap.cc:492] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.334038 1169 tablet_bootstrap.cc:492] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.334064 1171 ts_tablet_manager.cc:1397] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.028s user 0.008s sys 0.005s
I20250905 19:52:28.334264 1169 ts_tablet_manager.cc:1397] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.002s
W20250905 19:52:28.334338 984 leader_election.cc:343] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.334437 984 leader_election.cc:304] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.334858 1798 raft_consensus.cc:2747] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.335268 1169 raft_consensus.cc:357] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.335688 1169 raft_consensus.cc:383] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.335793 1169 raft_consensus.cc:738] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.336081 1169 consensus_queue.cc:260] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.334789 1171 raft_consensus.cc:357] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.336380 1171 raft_consensus.cc:383] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.336449 1171 raft_consensus.cc:738] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.336784 1169 ts_tablet_manager.cc:1428] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:28.337096 1169 tablet_bootstrap.cc:492] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.337826 1170 tablet_bootstrap.cc:492] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.338002 1170 ts_tablet_manager.cc:1397] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.008s
I20250905 19:52:28.338917 1170 raft_consensus.cc:357] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.339105 1170 raft_consensus.cc:383] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.339224 1170 raft_consensus.cc:738] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.339391 1169 tablet_bootstrap.cc:654] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.339478 1170 consensus_queue.cc:260] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.341035 1170 ts_tablet_manager.cc:1428] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:28.341504 1170 tablet_bootstrap.cc:492] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.342098 1171 consensus_queue.cc:260] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.303102 2205 ts_tablet_manager.cc:1918] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:28.342981 2205 log.cc:1199] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/813f208881ac4782874138987a710d93
I20250905 19:52:28.343593 2205 ts_tablet_manager.cc:1939] T 813f208881ac4782874138987a710d93 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:28.343897 1170 tablet_bootstrap.cc:654] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.344099 2206 ts_tablet_manager.cc:1918] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:28.344645 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:28.344923 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 2 got 0
I20250905 19:52:28.345103 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:28.345523 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:28.345722 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:28.345975 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:28.346277 2206 log.cc:1199] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/813f208881ac4782874138987a710d93
I20250905 19:52:28.346442 1170 tablet_bootstrap.cc:492] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.346307 1546 catalog_manager.cc:5582] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:28.346709 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:28.346891 1170 ts_tablet_manager.cc:1397] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20250905 19:52:28.347358 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:28.347636 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:28.348052 776 raft_consensus.cc:3058] T ff2f289c293f41379e4af470d747860c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.348340 1170 raft_consensus.cc:357] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.348534 1170 raft_consensus.cc:383] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.348601 1170 raft_consensus.cc:738] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.348912 1170 consensus_queue.cc:260] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.345350 1171 ts_tablet_manager.cc:1428] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:28.350440 2206 ts_tablet_manager.cc:1939] T 813f208881ac4782874138987a710d93 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:28.350452 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 813f208881ac4782874138987a710d93 (table default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]) successfully deleted
I20250905 19:52:28.350728 1171 tablet_bootstrap.cc:492] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.351554 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 813f208881ac4782874138987a710d93 (table default.Table_55ca8eccc0224578835d4f9f8fae2311 [id=d77f49f91179475c950609bd37ed39c7]) successfully deleted
I20250905 19:52:28.353601 1171 tablet_bootstrap.cc:654] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.354648 1170 ts_tablet_manager.cc:1428] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.002s sys 0.001s
I20250905 19:52:28.356813 1170 tablet_bootstrap.cc:492] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.361410 1170 tablet_bootstrap.cc:654] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.364751 2108 raft_consensus.cc:491] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.364912 2108 raft_consensus.cc:513] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.365505 2108 leader_election.cc:290] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.366176 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efceaf3c7bda44ddabd5d38130af3dd2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.366367 1050 raft_consensus.cc:2466] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:28.366827 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efceaf3c7bda44ddabd5d38130af3dd2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.366920 849 leader_election.cc:304] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:28.367285 848 leader_election.cc:343] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.367409 2095 raft_consensus.cc:2802] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.367468 2095 raft_consensus.cc:491] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.367527 2095 raft_consensus.cc:3058] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.368111 1798 consensus_queue.cc:1035] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.370059 2072 fs_manager.cc:714] Time spent opening directory manager: real 0.079s user 0.008s sys 0.004s
I20250905 19:52:28.370281 2095 raft_consensus.cc:513] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.371282 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efceaf3c7bda44ddabd5d38130af3dd2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:28.371452 1050 raft_consensus.cc:3058] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.371766 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efceaf3c7bda44ddabd5d38130af3dd2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:28.372246 848 leader_election.cc:343] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.373180 1171 tablet_bootstrap.cc:492] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.373304 1171 ts_tablet_manager.cc:1397] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.005s
I20250905 19:52:28.373960 1171 raft_consensus.cc:357] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.374115 1171 raft_consensus.cc:383] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.374166 1171 raft_consensus.cc:738] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.374145 1050 raft_consensus.cc:2466] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:28.374366 1171 consensus_queue.cc:260] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.374831 849 leader_election.cc:304] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.375094 2108 raft_consensus.cc:2802] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.375207 2108 raft_consensus.cc:695] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:28.375415 2108 consensus_queue.cc:237] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.376894 1849 raft_consensus.cc:491] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.377082 1849 raft_consensus.cc:513] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.377700 1849 leader_election.cc:290] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.378741 2095 leader_election.cc:290] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.378762 711 leader_election.cc:343] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:28.380416 1170 tablet_bootstrap.cc:492] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.380750 1170 ts_tablet_manager.cc:1397] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.002s
I20250905 19:52:28.381791 1170 raft_consensus.cc:357] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.382030 1170 raft_consensus.cc:383] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.382140 1170 raft_consensus.cc:738] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.382462 1170 consensus_queue.cc:260] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.382817 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5f076d9917f40d899d7422ad974fe0f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.383237 913 raft_consensus.cc:2466] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:28.383610 1170 ts_tablet_manager.cc:1428] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:28.383931 1170 tablet_bootstrap.cc:492] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.383893 712 leader_election.cc:304] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.384377 1849 raft_consensus.cc:2802] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.384483 1849 raft_consensus.cc:491] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.384579 1849 raft_consensus.cc:3058] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.385845 1170 tablet_bootstrap.cc:654] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.378857 1171 ts_tablet_manager.cc:1428] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250905 19:52:28.389101 1171 tablet_bootstrap.cc:492] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:28.389991 848 consensus_peers.cc:489] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.391856 1171 tablet_bootstrap.cc:654] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.392315 1849 raft_consensus.cc:513] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.396782 2100 consensus_queue.cc:1035] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.397725 774 raft_consensus.cc:3058] T ac2e51a9071c4e1dae9e147687291175 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.397990 2100 raft_consensus.cc:491] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.398175 2100 raft_consensus.cc:513] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.398898 2100 leader_election.cc:290] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.400120 711 leader_election.cc:343] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b
I20250905 19:52:28.400732 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ae83eea5f74e9187713e8b82c9a23b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.400940 911 raft_consensus.cc:2466] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:28.401698 712 leader_election.cc:304] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.402185 2100 raft_consensus.cc:2802] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.402281 2100 raft_consensus.cc:491] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.402381 2100 raft_consensus.cc:3058] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.404970 1169 tablet_bootstrap.cc:492] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.405180 1169 ts_tablet_manager.cc:1397] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.068s user 0.003s sys 0.005s
I20250905 19:52:28.405640 2100 raft_consensus.cc:513] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.406586 1169 raft_consensus.cc:357] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.406852 1169 raft_consensus.cc:383] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.406927 1169 raft_consensus.cc:738] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.407291 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "05ae83eea5f74e9187713e8b82c9a23b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:28.407461 911 raft_consensus.cc:3058] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.410712 911 raft_consensus.cc:2466] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:28.411355 712 leader_election.cc:304] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:28.411337 1169 consensus_queue.cc:260] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.412245 1169 ts_tablet_manager.cc:1428] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:28.412326 1584 consensus_queue.cc:1035] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.415516 2160 raft_consensus.cc:2802] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.415727 2160 raft_consensus.cc:695] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:28.416054 2160 consensus_queue.cc:237] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.421622 2272 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20250905 19:52:28.423454 1169 tablet_bootstrap.cc:492] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.426306 1584 consensus_queue.cc:1035] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.426923 2072 fs_manager.cc:730] Time spent opening block manager: real 0.052s user 0.004s sys 0.002s
I20250905 19:52:28.427148 2072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
uuid: "c401c9f3020b4ae5931c566c6bcb1711"
format_stamp: "Formatted at 2025-09-05 19:52:10 on dist-test-slave-b1r7"
I20250905 19:52:28.428227 2072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20250905 19:52:28.432962 1169 tablet_bootstrap.cc:654] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.434756 1849 leader_election.cc:290] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.435352 2100 leader_election.cc:290] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.435904 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5f076d9917f40d899d7422ad974fe0f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
W20250905 19:52:28.436046 711 leader_election.cc:343] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b
I20250905 19:52:28.436142 913 raft_consensus.cc:3058] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.436432 1171 tablet_bootstrap.cc:492] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:28.436439 711 leader_election.cc:343] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:28.436573 1171 ts_tablet_manager.cc:1397] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.013s
I20250905 19:52:28.437211 1171 raft_consensus.cc:357] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.437379 1171 raft_consensus.cc:383] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.437431 1171 raft_consensus.cc:738] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.437640 1171 consensus_queue.cc:260] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.438105 1171 ts_tablet_manager.cc:1428] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:28.438385 1171 tablet_bootstrap.cc:492] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.442167 913 raft_consensus.cc:2466] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:28.444185 712 leader_election.cc:304] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.444504 2160 raft_consensus.cc:2802] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.444661 2160 raft_consensus.cc:695] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:28.444967 2160 consensus_queue.cc:237] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.456653 1169 tablet_bootstrap.cc:492] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.456809 1169 ts_tablet_manager.cc:1397] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.000s
I20250905 19:52:28.458155 1169 raft_consensus.cc:357] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.458336 1169 raft_consensus.cc:383] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.458388 1169 raft_consensus.cc:738] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.458583 1169 consensus_queue.cc:260] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.459079 1169 ts_tablet_manager.cc:1428] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:28.459363 1169 tablet_bootstrap.cc:492] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.460641 774 raft_consensus.cc:3058] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.461139 1169 tablet_bootstrap.cc:654] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.464619 1171 tablet_bootstrap.cc:654] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.467164 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:28.467460 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:28.479511 1170 tablet_bootstrap.cc:492] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.479781 1170 ts_tablet_manager.cc:1397] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.096s user 0.002s sys 0.005s
I20250905 19:52:28.481012 1170 raft_consensus.cc:357] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.481319 1170 raft_consensus.cc:383] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.481429 1170 raft_consensus.cc:738] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.481731 1170 consensus_queue.cc:260] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.482437 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:28.482735 1170 ts_tablet_manager.cc:1428] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:28.483115 1170 tablet_bootstrap.cc:492] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.483182 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:28.483408 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:28.483664 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 2 got 1
I20250905 19:52:28.483985 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:28.486110 1170 tablet_bootstrap.cc:654] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.486183 1171 tablet_bootstrap.cc:492] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.486332 1171 ts_tablet_manager.cc:1397] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.048s user 0.007s sys 0.000s
I20250905 19:52:28.487007 1171 raft_consensus.cc:357] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.487180 1171 raft_consensus.cc:383] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.487233 1171 raft_consensus.cc:738] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.488698 1171 consensus_queue.cc:260] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.489351 1171 ts_tablet_manager.cc:1428] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:28.489711 2163 raft_consensus.cc:491] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.489862 2163 raft_consensus.cc:513] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.490546 2163 leader_election.cc:290] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.491803 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f52ff57fb1a4ea8a1ad4d5f0480c7f2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:28.492385 984 leader_election.cc:343] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.484652 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:28.495409 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:28.499387 1171 tablet_bootstrap.cc:492] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.500207 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:28.500727 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:28.501153 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:28.501557 1169 tablet_bootstrap.cc:492] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.501577 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:28.501710 1169 ts_tablet_manager.cc:1397] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.042s user 0.000s sys 0.006s
I20250905 19:52:28.501855 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:28.502332 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:28.502576 1169 raft_consensus.cc:357] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.502678 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:28.502760 1169 raft_consensus.cc:383] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.502821 1169 raft_consensus.cc:738] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.503049 1169 consensus_queue.cc:260] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.503635 1169 ts_tablet_manager.cc:1428] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:28.504498 1171 tablet_bootstrap.cc:654] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.508253 1169 tablet_bootstrap.cc:492] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.510587 1169 tablet_bootstrap.cc:654] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.511042 2163 raft_consensus.cc:491] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.511190 2163 raft_consensus.cc:513] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.511814 2163 leader_election.cc:290] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.512892 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.513062 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.513509 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:28.513605 986 leader_election.cc:343] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:28.513904 984 leader_election.cc:343] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.514019 984 leader_election.cc:304] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.514248 2163 raft_consensus.cc:2747] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.515890 2100 raft_consensus.cc:491] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.516139 2100 raft_consensus.cc:513] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.516794 2100 leader_election.cc:290] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.518189 712 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
W20250905 19:52:28.518837 711 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:28.518980 711 leader_election.cc:304] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.519369 2100 raft_consensus.cc:2747] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.520905 1169 tablet_bootstrap.cc:492] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.521042 1169 ts_tablet_manager.cc:1397] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.013s user 0.002s sys 0.002s
I20250905 19:52:28.521834 1169 raft_consensus.cc:357] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.522006 1169 raft_consensus.cc:383] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.522059 1169 raft_consensus.cc:738] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.522256 1169 consensus_queue.cc:260] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.522320 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:28.522657 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:28.522742 1169 ts_tablet_manager.cc:1428] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:28.523015 1169 tablet_bootstrap.cc:492] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.517495 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f52ff57fb1a4ea8a1ad4d5f0480c7f2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.524361 986 leader_election.cc:343] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.524477 986 leader_election.cc:304] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.524801 2163 raft_consensus.cc:2747] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.524792 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b2c7906a718c46d9a76587a70d96a1f9 (table default.Table_a82a3756e48a4bfc99f4aacf9b57f6d3 [id=99384aac4e624d6db630b249c3ba2543]). Expected version 2 got 1
I20250905 19:52:28.528254 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:28.528656 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:28.529569 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:28.529951 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:28.531081 1171 tablet_bootstrap.cc:492] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.531288 1171 ts_tablet_manager.cc:1397] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250905 19:52:28.531447 1170 tablet_bootstrap.cc:492] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.531620 1170 ts_tablet_manager.cc:1397] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.007s
I20250905 19:52:28.532285 1171 raft_consensus.cc:357] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.532527 1171 raft_consensus.cc:383] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.532464 1170 raft_consensus.cc:357] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.532656 1170 raft_consensus.cc:383] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.532680 1171 raft_consensus.cc:738] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.532752 1170 raft_consensus.cc:738] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.532814 1584 consensus_queue.cc:1035] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.533036 1170 consensus_queue.cc:260] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.533082 1171 consensus_queue.cc:260] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.534188 1171 ts_tablet_manager.cc:1428] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:28.537390 1169 tablet_bootstrap.cc:654] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.542411 1171 tablet_bootstrap.cc:492] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.545255 1171 tablet_bootstrap.cc:654] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.547464 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:28.547729 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:28.547902 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:28.548100 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:28.548307 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:28.548475 1170 ts_tablet_manager.cc:1428] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250905 19:52:28.548534 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
W20250905 19:52:28.549645 848 consensus_peers.cc:489] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.550069 2072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20250905 19:52:28.551172 1169 tablet_bootstrap.cc:492] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.551400 1169 ts_tablet_manager.cc:1397] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.003s sys 0.003s
I20250905 19:52:28.552332 1169 raft_consensus.cc:357] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.552539 1169 raft_consensus.cc:383] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.552603 1169 raft_consensus.cc:738] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.552819 1169 consensus_queue.cc:260] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.553318 1169 ts_tablet_manager.cc:1428] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:28.553620 1169 tablet_bootstrap.cc:492] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.554180 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:28.555325 1169 tablet_bootstrap.cc:654] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.556185 1170 tablet_bootstrap.cc:492] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.556736 1171 tablet_bootstrap.cc:492] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.556913 1171 ts_tablet_manager.cc:1397] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20250905 19:52:28.557403 2072 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20250905 19:52:28.557777 2072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20250905 19:52:28.557974 1170 tablet_bootstrap.cc:654] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.558269 1171 raft_consensus.cc:357] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.558455 1171 raft_consensus.cc:383] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.558581 1171 raft_consensus.cc:738] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.558840 1171 consensus_queue.cc:260] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.559358 1171 ts_tablet_manager.cc:1428] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:28.560452 1171 tablet_bootstrap.cc:492] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.562114 1171 tablet_bootstrap.cc:654] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.562755 2100 raft_consensus.cc:491] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.562925 2100 raft_consensus.cc:513] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.563611 2100 leader_election.cc:290] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.564186 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13d282e1dc54d1faa2882c9b39d17cb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.564360 914 raft_consensus.cc:2466] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
W20250905 19:52:28.564502 711 leader_election.cc:343] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:28.564790 712 leader_election.cc:304] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.565411 2100 raft_consensus.cc:2802] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.565506 2100 raft_consensus.cc:491] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.565609 2100 raft_consensus.cc:3058] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.566008 1171 tablet_bootstrap.cc:492] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.566134 1171 ts_tablet_manager.cc:1397] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.001s
I20250905 19:52:28.567080 1171 raft_consensus.cc:357] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.567281 1171 raft_consensus.cc:383] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.567339 1171 raft_consensus.cc:738] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.567595 1171 consensus_queue.cc:260] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.568876 2100 raft_consensus.cc:513] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.570463 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b13d282e1dc54d1faa2882c9b39d17cb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:28.570637 914 raft_consensus.cc:3058] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.573959 2100 leader_election.cc:290] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.576906 711 leader_election.cc:343] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:28.577323 1170 tablet_bootstrap.cc:492] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.577493 1170 ts_tablet_manager.cc:1397] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.000s
I20250905 19:52:28.578295 1170 raft_consensus.cc:357] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.578501 1170 raft_consensus.cc:383] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.578576 1170 raft_consensus.cc:738] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.578809 1170 consensus_queue.cc:260] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.579846 1170 ts_tablet_manager.cc:1428] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:28.549225 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:28.582568 1170 tablet_bootstrap.cc:492] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.582717 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 2 got 0
I20250905 19:52:28.582996 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:28.583280 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:28.583516 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:28.583681 1542 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:28.584976 1170 tablet_bootstrap.cc:654] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.586645 1171 ts_tablet_manager.cc:1428] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.020s user 0.001s sys 0.001s
I20250905 19:52:28.587951 1171 tablet_bootstrap.cc:492] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.589752 1171 tablet_bootstrap.cc:654] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.591590 1169 tablet_bootstrap.cc:492] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.592198 1169 ts_tablet_manager.cc:1397] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.000s
I20250905 19:52:28.594411 2109 consensus_queue.cc:1035] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.599104 2100 raft_consensus.cc:491] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.599426 2100 raft_consensus.cc:513] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.601539 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.601527 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9b3acf17eab64913b5b33677196a5781" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.602205 711 leader_election.cc:304] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.602538 2160 raft_consensus.cc:2747] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.603468 914 raft_consensus.cc:2466] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:28.604214 712 leader_election.cc:304] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.604754 2160 raft_consensus.cc:2802] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.604926 2160 raft_consensus.cc:695] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:28.605271 2160 consensus_queue.cc:237] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.612543 2100 leader_election.cc:290] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.615459 2163 raft_consensus.cc:491] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.615631 2163 raft_consensus.cc:513] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.616312 2163 leader_election.cc:290] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.617067 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.617247 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.617717 984 leader_election.cc:343] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:28.617971 986 leader_election.cc:343] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.618129 986 leader_election.cc:304] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.618602 2163 raft_consensus.cc:2747] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.618992 1973 raft_consensus.cc:491] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.619190 1973 raft_consensus.cc:513] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.620553 1973 leader_election.cc:290] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.621340 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:28.622562 848 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.623628 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.623854 1047 raft_consensus.cc:2466] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:28.623898 2100 raft_consensus.cc:491] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.624583 849 leader_election.cc:304] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.625070 2095 raft_consensus.cc:2802] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.625159 2095 raft_consensus.cc:491] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.625237 2095 raft_consensus.cc:3058] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.624859 2100 raft_consensus.cc:513] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.629206 711 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 19e4edddafcd4d22a0e0ae66b0b953e0
W20250905 19:52:28.629709 712 leader_election.cc:343] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:28.629669 2095 raft_consensus.cc:513] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.629868 712 leader_election.cc:304] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.630216 2154 raft_consensus.cc:2747] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.630453 2095 leader_election.cc:290] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.631341 848 consensus_peers.cc:489] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.631608 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:28.631815 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:28.632000 1046 raft_consensus.cc:3058] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:28.632126 848 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.632328 2100 leader_election.cc:290] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.635169 1046 raft_consensus.cc:2466] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:28.635996 849 leader_election.cc:304] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.636543 2095 raft_consensus.cc:2802] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.636833 2095 raft_consensus.cc:695] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:28.637217 2095 consensus_queue.cc:237] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.641810 848 consensus_peers.cc:489] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.643108 1170 tablet_bootstrap.cc:492] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.643390 1170 ts_tablet_manager.cc:1397] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.061s user 0.013s sys 0.009s
I20250905 19:52:28.644011 1658 raft_consensus.cc:491] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.644133 1658 raft_consensus.cc:513] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.644357 1170 raft_consensus.cc:357] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.644709 1658 leader_election.cc:290] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.645062 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b0d4be788f841c2860f5dd65275b623" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:28.649497 848 leader_election.cc:343] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.649675 1170 raft_consensus.cc:383] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.649650 1658 consensus_queue.cc:1035] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.649806 1170 raft_consensus.cc:738] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.650156 1170 consensus_queue.cc:260] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.651706 1170 ts_tablet_manager.cc:1428] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20250905 19:52:28.652158 1170 tablet_bootstrap.cc:492] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.656268 1170 tablet_bootstrap.cc:654] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:28.662914 848 consensus_peers.cc:489] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.676364 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b0d4be788f841c2860f5dd65275b623" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.676709 1045 raft_consensus.cc:2466] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:28.677264 849 leader_election.cc:304] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.677979 2095 raft_consensus.cc:2802] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.678113 2095 raft_consensus.cc:491] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.678221 2095 raft_consensus.cc:3058] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.688324 1170 tablet_bootstrap.cc:492] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.688570 1170 ts_tablet_manager.cc:1397] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.007s
I20250905 19:52:28.689745 1170 raft_consensus.cc:357] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.690070 1170 raft_consensus.cc:383] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.690167 1170 raft_consensus.cc:738] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.690428 1170 consensus_queue.cc:260] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.691184 1170 ts_tablet_manager.cc:1428] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20250905 19:52:28.691625 2100 raft_consensus.cc:491] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.691825 2100 raft_consensus.cc:513] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.692235 1170 tablet_bootstrap.cc:492] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.692580 2100 leader_election.cc:290] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.694075 1170 tablet_bootstrap.cc:654] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.698881 2163 raft_consensus.cc:491] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.699051 2163 raft_consensus.cc:513] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.699649 2163 leader_election.cc:290] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.700580 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3ea2e2059a4425892fa3c0757e69e4c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.700954 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.701057 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3ea2e2059a4425892fa3c0757e69e4c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.701611 984 leader_election.cc:343] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:28.702405 986 leader_election.cc:343] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.702561 986 leader_election.cc:304] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.703168 2163 raft_consensus.cc:2747] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.702418 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "baefaa1f042348c1874e306f1e5a8781" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.704210 711 leader_election.cc:304] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.705160 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:28.705421 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:28.706048 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:28.706703 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:28.707127 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 2 got 1
I20250905 19:52:28.707491 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:28.707633 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:28.707893 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:28.708676 2163 raft_consensus.cc:491] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.708901 2163 raft_consensus.cc:513] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.709800 2163 leader_election.cc:290] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.710837 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31e332cdfedd4ff6b5026baa73e2caf3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:28.711433 984 leader_election.cc:343] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.712402 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31e332cdfedd4ff6b5026baa73e2caf3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.716534 986 leader_election.cc:343] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.716720 986 leader_election.cc:304] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.717693 1798 raft_consensus.cc:2747] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.721426 2160 raft_consensus.cc:2747] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.722465 2007 consensus_queue.cc:1035] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:28.724751 848 consensus_peers.cc:489] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.729869 2160 raft_consensus.cc:491] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.730131 2160 raft_consensus.cc:513] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.731508 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.732805 2160 leader_election.cc:290] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.732834 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0039b8f0ba704907b2820c9e3a69cd73" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.733376 712 leader_election.cc:304] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.734375 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:28.734735 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:28.734720 2095 raft_consensus.cc:513] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.735006 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:28.735402 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:28.735802 2095 leader_election.cc:290] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.736086 2160 raft_consensus.cc:2747] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.736315 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b0d4be788f841c2860f5dd65275b623" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:28.736495 1046 raft_consensus.cc:3058] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.738488 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b0d4be788f841c2860f5dd65275b623" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:28.739022 848 consensus_peers.cc:489] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:28.739192 848 leader_election.cc:343] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.740037 776 raft_consensus.cc:3058] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.740612 2007 consensus_queue.cc:1035] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:28.747761 711 consensus_peers.cc:489] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.750931 1169 raft_consensus.cc:357] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.751309 1169 raft_consensus.cc:383] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.751415 1169 raft_consensus.cc:738] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.751662 1169 consensus_queue.cc:260] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.752698 1169 ts_tablet_manager.cc:1428] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.160s user 0.001s sys 0.001s
I20250905 19:52:28.753082 1046 raft_consensus.cc:2466] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:28.753911 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:28.754463 849 leader_election.cc:304] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.755043 2007 raft_consensus.cc:2802] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.755159 2007 raft_consensus.cc:695] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:28.755409 2007 consensus_queue.cc:237] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.757967 774 raft_consensus.cc:3058] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.761801 1536 catalog_manager.cc:5582] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:28.762265 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
W20250905 19:52:28.763615 848 consensus_peers.cc:489] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:28.764011 711 consensus_peers.cc:489] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0267ac3e3f1a45978807737eca5d91ef. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.766209 2160 consensus_queue.cc:1035] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.772648 1170 tablet_bootstrap.cc:492] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.772883 1170 ts_tablet_manager.cc:1397] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.081s user 0.008s sys 0.020s
I20250905 19:52:28.773924 1170 raft_consensus.cc:357] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.774171 1170 raft_consensus.cc:383] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.774259 1170 raft_consensus.cc:738] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.775434 1170 consensus_queue.cc:260] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.776330 1170 ts_tablet_manager.cc:1428] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250905 19:52:28.776753 1170 tablet_bootstrap.cc:492] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.777094 2160 consensus_queue.cc:1035] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.779302 1170 tablet_bootstrap.cc:654] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.788290 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:28.788916 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:28.790350 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:28.790998 774 raft_consensus.cc:3058] T 9b3acf17eab64913b5b33677196a5781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.791152 1798 raft_consensus.cc:491] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.791311 1798 raft_consensus.cc:513] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.791800 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:28.792059 1798 leader_election.cc:290] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.795563 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:28.796254 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd9380e8ffc432da4a684934c2df68f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.796780 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd9380e8ffc432da4a684934c2df68f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.797138 2160 raft_consensus.cc:491] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:28.797348 986 leader_election.cc:343] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.797464 2160 raft_consensus.cc:513] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.797818 984 leader_election.cc:343] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.798128 984 leader_election.cc:304] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.798384 1798 raft_consensus.cc:2747] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.798516 2160 leader_election.cc:290] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.799487 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:28.800729 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b2c7906a718c46d9a76587a70d96a1f9 (table default.Table_a82a3756e48a4bfc99f4aacf9b57f6d3 [id=99384aac4e624d6db630b249c3ba2543]). Expected version 2 got 1
I20250905 19:52:28.800675 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9630f1feb6d74472b5f6f4b54151311c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.800915 914 raft_consensus.cc:2466] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:28.801469 712 leader_election.cc:304] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:28.801832 2160 raft_consensus.cc:2802] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.801990 2160 raft_consensus.cc:491] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.802145 2160 raft_consensus.cc:3058] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:28.804065 711 leader_election.cc:343] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:28.805435 2160 raft_consensus.cc:513] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.806886 711 leader_election.cc:343] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:28.807168 2160 leader_election.cc:290] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.808504 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:28.809988 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:28.810562 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:28.811124 1536 catalog_manager.cc:5582] T 9b3acf17eab64913b5b33677196a5781 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:28.814086 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9630f1feb6d74472b5f6f4b54151311c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:28.814335 914 raft_consensus.cc:3058] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.815210 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:28.818136 914 raft_consensus.cc:2466] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:28.818737 712 leader_election.cc:304] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:28.819350 2160 raft_consensus.cc:2802] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.820026 2160 raft_consensus.cc:695] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:28.820416 2160 consensus_queue.cc:237] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.824473 2100 raft_consensus.cc:491] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.824637 2100 raft_consensus.cc:513] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.825300 2100 leader_election.cc:290] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.825731 2095 consensus_queue.cc:1035] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:28.826768 712 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
W20250905 19:52:28.827703 711 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:28.827854 711 leader_election.cc:304] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.828588 2100 raft_consensus.cc:2747] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.831293 1536 catalog_manager.cc:5582] T 4a518797d26140f5b9abe41eab415040 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:28.832257 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:28.832605 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:28.832285 2320 tablet.cc:1722] T 415508e72ee34bc391c4369b5f8a2f48 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:28.753927 1169 tablet_bootstrap.cc:492] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.834723 1171 tablet_bootstrap.cc:492] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:28.834873 1171 ts_tablet_manager.cc:1397] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.248s user 0.033s sys 0.095s
I20250905 19:52:28.835811 1171 raft_consensus.cc:357] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.836326 1171 raft_consensus.cc:383] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.836395 1171 raft_consensus.cc:738] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.836711 1171 consensus_queue.cc:260] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.837126 1171 ts_tablet_manager.cc:1428] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:28.837394 1171 tablet_bootstrap.cc:492] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:28.839428 1171 tablet_bootstrap.cc:654] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.841745 1849 raft_consensus.cc:491] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.841917 1849 raft_consensus.cc:513] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.842548 1849 leader_election.cc:290] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:28.843837 712 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
W20250905 19:52:28.844949 711 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:28.845077 711 leader_election.cc:304] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.845603 1849 raft_consensus.cc:2747] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.845937 1169 tablet_bootstrap.cc:654] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.847417 1536 catalog_manager.cc:5582] T de19c897d3314fecb17cedb3370eed93 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:28.848192 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:28.848649 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:28.848974 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 2 got 0
I20250905 19:52:28.849140 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:28.849383 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:28.849571 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:28.849915 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
W20250905 19:52:28.852443 984 consensus_peers.cc:489] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.862035 1170 tablet_bootstrap.cc:492] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.862258 1170 ts_tablet_manager.cc:1397] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.086s user 0.008s sys 0.005s
I20250905 19:52:28.863034 1170 raft_consensus.cc:357] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.863262 1170 raft_consensus.cc:383] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.863338 1170 raft_consensus.cc:738] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.863574 1170 consensus_queue.cc:260] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.864080 1170 ts_tablet_manager.cc:1428] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250905 19:52:28.865689 1170 tablet_bootstrap.cc:492] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.865568 1169 tablet_bootstrap.cc:492] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.866288 1169 ts_tablet_manager.cc:1397] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.000s
I20250905 19:52:28.867122 1169 raft_consensus.cc:357] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.867300 1169 raft_consensus.cc:383] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.867344 1170 tablet_bootstrap.cc:654] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.867353 1169 raft_consensus.cc:738] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.867543 1169 consensus_queue.cc:260] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.868213 1169 ts_tablet_manager.cc:1428] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:28.870752 1536 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:28.872130 1169 tablet_bootstrap.cc:492] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.874068 1169 tablet_bootstrap.cc:654] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.878983 1798 raft_consensus.cc:491] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.879154 1798 raft_consensus.cc:513] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.879822 1798 leader_election.cc:290] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.880988 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf448f9abc348df8d2d2501420d8f45" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.881366 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf448f9abc348df8d2d2501420d8f45" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.881595 984 leader_election.cc:343] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:28.881974 986 leader_election.cc:343] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.882090 986 leader_election.cc:304] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.882328 1798 raft_consensus.cc:2747] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.882443 2007 consensus_queue.cc:1035] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.883371 1798 raft_consensus.cc:491] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.883526 1798 raft_consensus.cc:513] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.884238 1798 leader_election.cc:290] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.885723 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc95a83ca872495894ef439cdda4e364" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:28.886296 986 leader_election.cc:343] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.886413 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc95a83ca872495894ef439cdda4e364" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:28.887416 984 leader_election.cc:343] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.887519 984 leader_election.cc:304] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.887876 1798 raft_consensus.cc:2747] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.892714 2007 raft_consensus.cc:491] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.892942 2007 raft_consensus.cc:513] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.893775 2007 leader_election.cc:290] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.894366 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:28.894555 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.894594 1047 raft_consensus.cc:2466] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:28.895112 848 leader_election.cc:343] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.895306 849 leader_election.cc:304] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.895829 2007 raft_consensus.cc:2802] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:28.895943 2007 raft_consensus.cc:491] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:28.896062 2007 raft_consensus.cc:3058] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.899978 2007 raft_consensus.cc:513] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.901129 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:28.902326 848 leader_election.cc:343] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:28.903409 711 consensus_peers.cc:489] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 78076f6534814939ae8c2db9a4bd1879. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:28.903003 2007 leader_election.cc:290] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.903532 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:28.903764 1047 raft_consensus.cc:3058] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.904022 2160 consensus_queue.cc:1035] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.906706 1047 raft_consensus.cc:2466] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:28.907322 849 leader_election.cc:304] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.907656 2007 raft_consensus.cc:2802] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:28.908241 2007 raft_consensus.cc:695] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:28.909011 2007 consensus_queue.cc:237] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.922472 2007 consensus_queue.cc:1035] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.926618 1169 tablet_bootstrap.cc:492] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.926764 1169 ts_tablet_manager.cc:1397] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.055s user 0.015s sys 0.007s
I20250905 19:52:28.928032 1169 raft_consensus.cc:357] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.928205 1169 raft_consensus.cc:383] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.928257 1169 raft_consensus.cc:738] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.928445 1169 consensus_queue.cc:260] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.929198 1170 tablet_bootstrap.cc:492] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.929395 1170 ts_tablet_manager.cc:1397] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.064s user 0.002s sys 0.009s
I20250905 19:52:28.929910 1169 ts_tablet_manager.cc:1428] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:28.930191 1169 tablet_bootstrap.cc:492] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.930267 1170 raft_consensus.cc:357] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.930450 1170 raft_consensus.cc:383] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.930532 1170 raft_consensus.cc:738] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.930732 1170 consensus_queue.cc:260] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.932886 1169 tablet_bootstrap.cc:654] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.933363 1170 ts_tablet_manager.cc:1428] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20250905 19:52:28.934129 2335 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:28.934357 1170 tablet_bootstrap.cc:492] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:28.934330 2335 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:28.937412 2335 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:28.938301 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 623 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:28.944458 1169 tablet_bootstrap.cc:492] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.944612 1169 ts_tablet_manager.cc:1397] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.000s
I20250905 19:52:28.945590 2100 raft_consensus.cc:491] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.945755 2100 raft_consensus.cc:513] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.946516 2100 leader_election.cc:290] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:28.946909 1169 raft_consensus.cc:357] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.947115 1169 raft_consensus.cc:383] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.947181 1169 raft_consensus.cc:738] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.947388 1169 consensus_queue.cc:260] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.948093 712 leader_election.cc:343] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 757a7d3fba8d43dda36474539b7296ba
W20250905 19:52:28.949281 711 leader_election.cc:343] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:28.949412 711 leader_election.cc:304] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:28.949553 1169 ts_tablet_manager.cc:1428] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:28.949807 1169 tablet_bootstrap.cc:492] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.950093 2100 raft_consensus.cc:2747] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.951385 1169 tablet_bootstrap.cc:654] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.952733 1170 tablet_bootstrap.cc:654] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.956794 1169 tablet_bootstrap.cc:492] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:28.956924 1169 ts_tablet_manager.cc:1397] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.003s
I20250905 19:52:28.957649 1169 raft_consensus.cc:357] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.959367 1169 raft_consensus.cc:383] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.959532 1973 consensus_queue.cc:1035] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.959441 1169 raft_consensus.cc:738] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.960055 1169 consensus_queue.cc:260] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.961207 1169 ts_tablet_manager.cc:1428] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:28.961521 1169 tablet_bootstrap.cc:492] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:28.963588 1169 tablet_bootstrap.cc:654] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.964385 776 raft_consensus.cc:3058] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:28.964828 2163 consensus_queue.cc:1035] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:28.973598 1170 tablet_bootstrap.cc:492] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:28.973901 1170 ts_tablet_manager.cc:1397] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.003s
I20250905 19:52:28.974983 1170 raft_consensus.cc:357] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.975940 2163 raft_consensus.cc:491] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:28.976151 2163 raft_consensus.cc:513] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:28.976974 2163 leader_election.cc:290] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:28.977421 1170 raft_consensus.cc:383] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:28.977536 1170 raft_consensus.cc:738] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:28.977581 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:28.978328 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:28.977794 1170 consensus_queue.cc:260] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:28.979483 986 leader_election.cc:343] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.979780 1170 ts_tablet_manager.cc:1428] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:28.981670 1170 tablet_bootstrap.cc:492] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:28.983806 984 leader_election.cc:343] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:28.983981 984 leader_election.cc:304] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:28.984597 2163 raft_consensus.cc:2747] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:28.989001 1170 tablet_bootstrap.cc:654] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:28.998430 1170 tablet_bootstrap.cc:492] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.001168 1170 ts_tablet_manager.cc:1397] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.004s
I20250905 19:52:29.002563 1170 raft_consensus.cc:357] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.002803 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e63b27ff4e434d76a8280f90c22f4082 consensus state has no leader
I20250905 19:52:29.002912 1170 raft_consensus.cc:383] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.003046 1170 raft_consensus.cc:738] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.003377 1170 consensus_queue.cc:260] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.004061 1171 tablet_bootstrap.cc:492] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.004252 1171 ts_tablet_manager.cc:1397] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.167s user 0.006s sys 0.006s
I20250905 19:52:29.004531 1170 ts_tablet_manager.cc:1428] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:29.005043 1170 tablet_bootstrap.cc:492] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.005116 1171 raft_consensus.cc:357] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.005373 1171 raft_consensus.cc:383] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.005462 1171 raft_consensus.cc:738] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.005683 1171 consensus_queue.cc:260] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.006261 1171 ts_tablet_manager.cc:1428] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:29.006582 1171 tablet_bootstrap.cc:492] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.010504 1170 tablet_bootstrap.cc:654] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.012401 1171 tablet_bootstrap.cc:654] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.022680 1170 tablet_bootstrap.cc:492] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.023556 1170 ts_tablet_manager.cc:1397] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.002s
I20250905 19:52:29.025682 1171 tablet_bootstrap.cc:492] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.025921 1171 ts_tablet_manager.cc:1397] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.002s
I20250905 19:52:29.027012 1171 raft_consensus.cc:357] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.027381 1171 raft_consensus.cc:383] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.027513 1171 raft_consensus.cc:738] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.027858 1171 consensus_queue.cc:260] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.028867 1171 ts_tablet_manager.cc:1428] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:29.029247 1171 tablet_bootstrap.cc:492] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:29.029908 511 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.62:38687, user_credentials={real_user=slave}} blocked reactor thread for 90347.5us
I20250905 19:52:29.031853 1171 tablet_bootstrap.cc:654] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.024647 1170 raft_consensus.cc:357] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.035825 1170 raft_consensus.cc:383] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.036365 1170 raft_consensus.cc:738] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.036986 1170 consensus_queue.cc:260] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.038429 511 leader_election.cc:336] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Network error: Client connection negotiation failed: client connection to 127.0.104.62:38687: connect: Connection refused (error 111)
I20250905 19:52:29.039108 511 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:29.039176 1170 ts_tablet_manager.cc:1428] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250905 19:52:29.040951 1170 tablet_bootstrap.cc:492] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.043771 2335 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:52:29.044531 1170 tablet_bootstrap.cc:654] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.050992 1171 tablet_bootstrap.cc:492] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.051200 1171 ts_tablet_manager.cc:1397] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.003s
I20250905 19:52:29.056339 2108 raft_consensus.cc:491] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.056506 2108 raft_consensus.cc:513] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.057127 2108 leader_election.cc:290] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.057855 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.058022 1047 raft_consensus.cc:2466] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.058545 849 leader_election.cc:304] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:29.058919 2007 raft_consensus.cc:2802] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.059005 2007 raft_consensus.cc:491] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.059087 2007 raft_consensus.cc:3058] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.061111 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.061635 848 leader_election.cc:343] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.061779 2007 raft_consensus.cc:513] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.062803 2007 leader_election.cc:290] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.063203 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.063426 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.063594 1047 raft_consensus.cc:3058] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.063637 1171 raft_consensus.cc:357] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.063704 848 leader_election.cc:343] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.063796 1171 raft_consensus.cc:383] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.063875 1171 raft_consensus.cc:738] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.065178 2100 consensus_queue.cc:1035] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.066267 1171 consensus_queue.cc:260] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.066429 1047 raft_consensus.cc:2466] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.066896 1171 ts_tablet_manager.cc:1428] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250905 19:52:29.067035 849 leader_election.cc:304] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.067269 1171 tablet_bootstrap.cc:492] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.067404 2108 raft_consensus.cc:2802] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.067519 2108 raft_consensus.cc:695] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:29.067744 2108 consensus_queue.cc:237] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.068214 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:29.068536 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:29.073632 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:29.075287 1171 tablet_bootstrap.cc:654] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.076402 1169 tablet_bootstrap.cc:492] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.076578 1169 ts_tablet_manager.cc:1397] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.115s user 0.009s sys 0.004s
I20250905 19:52:29.076725 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:29.077463 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:29.077445 1169 raft_consensus.cc:357] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.077626 1169 raft_consensus.cc:383] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.077683 1169 raft_consensus.cc:738] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.077920 1169 consensus_queue.cc:260] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.078434 1169 ts_tablet_manager.cc:1428] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:29.078013 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:29.078730 1169 tablet_bootstrap.cc:492] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.078742 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:29.080053 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:29.080353 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]). Expected version 2 got 1
I20250905 19:52:29.080739 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
W20250905 19:52:29.081167 711 consensus_peers.cc:489] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.081349 1169 tablet_bootstrap.cc:654] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.083338 1170 tablet_bootstrap.cc:492] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.083562 1170 ts_tablet_manager.cc:1397] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.043s user 0.002s sys 0.003s
I20250905 19:52:29.084352 1170 raft_consensus.cc:357] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.081753 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:29.085664 1170 raft_consensus.cc:383] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.085681 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:29.086293 1170 raft_consensus.cc:738] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.086591 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:29.086791 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:29.086992 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:29.087270 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:29.089041 2160 raft_consensus.cc:491] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.089265 2160 raft_consensus.cc:513] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.090541 2160 leader_election.cc:290] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:29.091689 711 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 4c76077e4adf45a7adc736b0e69032eb
W20250905 19:52:29.091877 712 leader_election.cc:343] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:29.092065 712 leader_election.cc:304] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.092671 2160 raft_consensus.cc:2747] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.092866 2108 consensus_queue.cc:1035] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.093334 1171 tablet_bootstrap.cc:492] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.093483 1171 ts_tablet_manager.cc:1397] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.002s
I20250905 19:52:29.094568 1171 raft_consensus.cc:357] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.094763 1171 raft_consensus.cc:383] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.094864 1171 raft_consensus.cc:738] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.095340 1171 consensus_queue.cc:260] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.095598 1169 tablet_bootstrap.cc:492] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.095733 1169 ts_tablet_manager.cc:1397] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.003s
I20250905 19:52:29.096194 1171 ts_tablet_manager.cc:1428] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:29.096534 1171 tablet_bootstrap.cc:492] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.097146 1169 raft_consensus.cc:357] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.097379 1169 raft_consensus.cc:383] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.097434 1169 raft_consensus.cc:738] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.097615 1169 consensus_queue.cc:260] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.098062 1169 ts_tablet_manager.cc:1428] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:29.098299 1169 tablet_bootstrap.cc:492] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.099617 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:29.101080 1169 tablet_bootstrap.cc:654] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.102254 1171 tablet_bootstrap.cc:654] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.102687 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:29.103295 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:29.104628 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:29.107686 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:29.108103 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:29.108881 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:29.109228 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:29.109583 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:29.109972 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:29.110112 2160 raft_consensus.cc:491] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.110282 2160 raft_consensus.cc:513] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.111137 2160 leader_election.cc:290] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.111995 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4edfa9a656554e02b6649e8430c215fb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.112208 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:29.112419 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:29.112573 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:29.112767 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:29.113867 1171 tablet_bootstrap.cc:492] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.114075 1171 ts_tablet_manager.cc:1397] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250905 19:52:29.114952 1171 raft_consensus.cc:357] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.115223 1171 raft_consensus.cc:383] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.115370 1171 raft_consensus.cc:738] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.115682 1171 consensus_queue.cc:260] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.116027 2160 raft_consensus.cc:491] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.116320 1171 ts_tablet_manager.cc:1428] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:29.116261 2160 raft_consensus.cc:513] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.116583 1171 tablet_bootstrap.cc:492] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.117054 2160 leader_election.cc:290] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.118242 1171 tablet_bootstrap.cc:654] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.118191 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.119263 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
W20250905 19:52:29.119804 848 consensus_peers.cc:489] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.120330 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4edfa9a656554e02b6649e8430c215fb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.120458 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a5af8232d31240ff855b79dcea29396b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.120149 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:29.120826 711 leader_election.cc:304] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.121495 2160 raft_consensus.cc:2747] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.122148 711 leader_election.cc:304] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.122706 2160 raft_consensus.cc:2747] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.125416 2108 raft_consensus.cc:491] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.125562 2108 raft_consensus.cc:513] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.126147 2108 leader_election.cc:290] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.126668 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:29.126772 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.127401 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07]). Expected version 2 got 0
I20250905 19:52:29.127389 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:29.127437 848 leader_election.cc:343] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.127568 1047 raft_consensus.cc:2466] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.127911 849 leader_election.cc:304] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:29.128095 711 consensus_peers.cc:489] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.128311 2108 raft_consensus.cc:2802] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.128407 2108 raft_consensus.cc:491] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.128420 1169 tablet_bootstrap.cc:492] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.128495 2108 raft_consensus.cc:3058] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.128532 1169 ts_tablet_manager.cc:1397] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.030s user 0.001s sys 0.002s
I20250905 19:52:29.128684 1543 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:29.129278 1169 raft_consensus.cc:357] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.129434 1169 raft_consensus.cc:383] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.129479 1169 raft_consensus.cc:738] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.129655 1169 consensus_queue.cc:260] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.129914 1540 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 6bb79e3126074736a23b128a8e14fab6
I20250905 19:52:29.130142 1169 ts_tablet_manager.cc:1428] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:29.130264 776 raft_consensus.cc:3058] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.130489 1540 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 32) took 3038 ms. Trace:
I20250905 19:52:29.130705 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 6bb79e3126074736a23b128a8e14fab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.130575 1540 rpcz_store.cc:276] 0905 19:52:26.091540 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:28.080514 (+1988974us) service_pool.cc:225] Handling call
0905 19:52:28.081435 (+ 921us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:28.081621 (+ 186us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:28.081703 (+ 82us) catalog_manager.cc:2799] Locking tablets
0905 19:52:28.131072 (+ 49369us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:28.168127 (+ 37055us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:28.271868 (+103741us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:29.129868 (+858000us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:29.130433 (+ 565us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:28.161932 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:28.162141 (+ 209us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:28.162151 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:28.162156 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:28.162350 (+ 194us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:28.162372 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:28.162381 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:28.162536 (+ 155us) tablet.cc:699] Row locks acquired
0905 19:52:28.162541 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:28.162591 (+ 50us) write_op.cc:270] Start()
0905 19:52:28.162639 (+ 48us) write_op.cc:276] Timestamp: P: 1757101948162564 usec, L: 0
0905 19:52:28.162645 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:28.162907 (+ 262us) log.cc:844] Serialized 725 byte log entry
0905 19:52:28.166866 (+ 3959us) op_driver.cc:464] REPLICATION: finished
0905 19:52:28.167007 (+ 141us) write_op.cc:301] APPLY: starting
0905 19:52:28.167025 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:28.167078 (+ 53us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:28.167105 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:28.167173 (+ 68us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:28.167206 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:28.167215 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:28.167293 (+ 78us) log.cc:844] Serialized 29 byte log entry
0905 19:52:28.167352 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.167372 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:28.167379 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:28.167445 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.167449 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":91,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":428,"num_ops":2,"prepare.queue_time_us":29993,"prepare.run_cpu_time_us":1143,"prepare.run_wall_time_us":1144,"raft.queue_time_us":415,"raft.run_cpu_time_us":292,"raft.run_wall_time_us":291,"replication_time_us":4176}]]}
I20250905 19:52:29.131314 2108 raft_consensus.cc:513] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.131927 2108 leader_election.cc:290] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.132511 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.132652 1050 raft_consensus.cc:3058] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.132550 1170 consensus_queue.cc:260] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.133136 1170 ts_tablet_manager.cc:1428] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.049s user 0.001s sys 0.000s
I20250905 19:52:29.133481 1170 tablet_bootstrap.cc:492] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.135172 1050 raft_consensus.cc:2466] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.135618 849 leader_election.cc:304] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:29.135911 2108 raft_consensus.cc:2802] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.136065 2108 raft_consensus.cc:695] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:29.136343 2108 consensus_queue.cc:237] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.138911 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 6bb79e3126074736a23b128a8e14fab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:29.147593 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6bb79e3126074736a23b128a8e14fab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.149058 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:29.149868 848 leader_election.cc:343] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.151156 1170 tablet_bootstrap.cc:654] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.152580 1171 tablet_bootstrap.cc:492] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.152763 1171 ts_tablet_manager.cc:1397] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.001s
I20250905 19:52:29.153131 1169 tablet_bootstrap.cc:492] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.153708 1171 raft_consensus.cc:357] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.154680 1539 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:29.155164 1169 tablet_bootstrap.cc:654] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.155402 1539 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 34) took 3063 ms. Trace:
I20250905 19:52:29.155519 1539 rpcz_store.cc:276] 0905 19:52:26.091703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:28.087560 (+1995857us) service_pool.cc:225] Handling call
0905 19:52:28.088203 (+ 643us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:28.088344 (+ 141us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:28.088402 (+ 58us) catalog_manager.cc:2799] Locking tablets
0905 19:52:28.362871 (+274469us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:28.380346 (+ 17475us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:28.411267 (+ 30921us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:29.154613 (+743346us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:29.155329 (+ 716us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:28.363833 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:28.364009 (+ 176us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:28.364018 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:28.364020 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:28.364211 (+ 191us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:28.364229 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:28.364236 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:28.364369 (+ 133us) tablet.cc:699] Row locks acquired
0905 19:52:28.364372 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:28.364420 (+ 48us) write_op.cc:270] Start()
0905 19:52:28.364488 (+ 68us) write_op.cc:276] Timestamp: P: 1757101948364393 usec, L: 0
0905 19:52:28.364493 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:28.364722 (+ 229us) log.cc:844] Serialized 725 byte log entry
0905 19:52:28.379423 (+ 14701us) op_driver.cc:464] REPLICATION: finished
0905 19:52:28.379577 (+ 154us) write_op.cc:301] APPLY: starting
0905 19:52:28.379594 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:28.379643 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:28.379669 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:28.379738 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:28.379772 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:28.379779 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:28.379856 (+ 77us) log.cc:844] Serialized 29 byte log entry
0905 19:52:28.379923 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.379951 (+ 28us) write_op.cc:454] Released schema lock
0905 19:52:28.379958 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:28.380051 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.380056 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":466,"num_ops":2,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":1015,"prepare.run_wall_time_us":1015,"raft.queue_time_us":6746,"raft.run_cpu_time_us":282,"raft.run_wall_time_us":284,"replication_time_us":14895}]]}
W20250905 19:52:29.156890 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6597c91bec4f4143ab3ed40afe53a35e consensus state has no leader
W20250905 19:52:29.157064 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9f953a7490434bc7afa5510c33e79401 consensus state has no leader
W20250905 19:52:29.158521 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet efb70327f55844879f89449a7116010f consensus state has no leader
W20250905 19:52:29.159031 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e10b4e82c4bc429b9a42db39c03a5267 consensus state has no leader
W20250905 19:52:29.162351 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6bb79e3126074736a23b128a8e14fab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bb79e3126074736a23b128a8e14fab6 already in progress: creating tablet
I20250905 19:52:29.166576 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 2350bd08f977424ba94ee8fa01f66efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:39172
W20250905 19:52:29.175729 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet d011d1ec8f634386872e7ab257de3187 consensus state has no leader
I20250905 19:52:29.178673 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.178687 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.179355 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.180158 2376 tablet_replica.cc:331] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:29.180428 2376 raft_consensus.cc:2241] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.181967 2376 raft_consensus.cc:2270] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.182716 2376 ts_tablet_manager.cc:1905] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.182653 2095 consensus_queue.cc:1035] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.183023 1171 raft_consensus.cc:383] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.183130 1171 raft_consensus.cc:738] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.183377 1171 consensus_queue.cc:260] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.184458 2378 tablet_replica.cc:331] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:29.184850 2378 raft_consensus.cc:2241] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.185004 2378 raft_consensus.cc:2270] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:29.184971 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6bb79e3126074736a23b128a8e14fab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bb79e3126074736a23b128a8e14fab6 already in progress: creating tablet
I20250905 19:52:29.185379 1171 ts_tablet_manager.cc:1428] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.032s user 0.003s sys 0.000s
I20250905 19:52:29.185839 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.185937 2378 ts_tablet_manager.cc:1905] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.187919 890 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.188455 891 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.190330 2100 consensus_queue.cc:1035] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.039s
I20250905 19:52:29.191761 1171 tablet_bootstrap.cc:492] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.193711 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.200757 2378 ts_tablet_manager.cc:1918] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.202000 1171 tablet_bootstrap.cc:654] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.206849 2108 consensus_queue.cc:1035] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
W20250905 19:52:29.210992 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet b3ea2e2059a4425892fa3c0757e69e4c consensus state has no leader
W20250905 19:52:29.212065 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet efceaf3c7bda44ddabd5d38130af3dd2 consensus state has no leader
I20250905 19:52:29.216791 2378 log.cc:1199] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:29.217547 2378 ts_tablet_manager.cc:1939] T 2350bd08f977424ba94ee8fa01f66efa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:29.218223 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 040e65fdb1104b07b953044f9b6b538a consensus state has no leader
I20250905 19:52:29.219890 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 2350bd08f977424ba94ee8fa01f66efa (table default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]) successfully deleted
I20250905 19:52:29.223125 1169 tablet_bootstrap.cc:492] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.223304 1169 ts_tablet_manager.cc:1397] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.070s user 0.005s sys 0.000s
I20250905 19:52:29.224300 1169 raft_consensus.cc:357] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.224514 1169 raft_consensus.cc:383] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.224572 1169 raft_consensus.cc:738] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.224833 1169 consensus_queue.cc:260] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.225174 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet e58bbb6575fd4437b242032a281914ea consensus state has no leader
W20250905 19:52:29.226224 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 6221d7d058944bdb9aa730791cf2417b consensus state has no leader
W20250905 19:52:29.219937 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet a860ab3f5ce54bd2b75bb71730a3c6e6 consensus state has no leader
W20250905 19:52:29.232015 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet af10abf0a5b34b3bbfea20e5c2005006 consensus state has no leader
W20250905 19:52:29.232920 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348], current schema version=2): Not found: failed to reset TS proxy: No leader found: Tablet 3fe2b75a3e59437c885abc18614f1da2 consensus state has no leader
W20250905 19:52:29.233307 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ae6fe4620dda4b1296416e01f8ffee29 consensus state has no leader
W20250905 19:52:29.235841 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet ad0760dfee934aacbd4ab1fb81e116c7 consensus state has no leader
W20250905 19:52:29.241353 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151], current schema version=3): Not found: failed to reset TS proxy: No leader found: Tablet ea6df8634a314ceabcf3fd004bf7e241 consensus state has no leader
I20250905 19:52:29.242655 1169 ts_tablet_manager.cc:1428] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.019s user 0.003s sys 0.000s
I20250905 19:52:29.246250 1169 tablet_bootstrap.cc:492] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.248168 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 2350bd08f977424ba94ee8fa01f66efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.248441 1169 tablet_bootstrap.cc:654] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.256716 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.258558 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.262642 848 consensus_peers.cc:489] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.264429 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.264659 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.265136 892 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.265427 880 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.265676 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.265722 2160 raft_consensus.cc:491] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.265908 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.265901 2160 raft_consensus.cc:513] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.266113 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.266305 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.266474 893 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.266672 884 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.266677 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.266882 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267051 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267228 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267468 887 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267659 890 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267763 892 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267946 884 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.268031 885 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.268172 880 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.268229 888 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.268381 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.267685 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.269080 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:29.269174 848 consensus_peers.cc:489] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.269328 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.269837 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 2350bd08f977424ba94ee8fa01f66efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.270602 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.270723 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.271008 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.272203 1798 raft_consensus.cc:491] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.272349 1798 raft_consensus.cc:513] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.272789 2160 leader_election.cc:290] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.272956 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.273110 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.273267 2007 raft_consensus.cc:491] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.274351 1798 leader_election.cc:290] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.275882 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.276023 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:29.276403 986 leader_election.cc:343] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.281132 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.281450 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.282022 2379 tablet_replica.cc:331] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:29.282079 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.281992 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.282330 748 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.283886 984 leader_election.cc:343] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.284066 984 leader_election.cc:304] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.284747 2386 raft_consensus.cc:2747] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.285920 741 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.286422 2007 raft_consensus.cc:513] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.287042 2007 leader_election.cc:290] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.287789 878 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.287873 2379 raft_consensus.cc:2241] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.288134 2379 raft_consensus.cc:2270] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.288846 2379 ts_tablet_manager.cc:1905] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.287846 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb72fe6049894527917e1481bcb7b03a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.295434 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb72fe6049894527917e1481bcb7b03a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.295701 1045 raft_consensus.cc:2466] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.302656 748 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.303618 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.306422 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.306641 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.306766 748 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.306879 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.306836 848 leader_election.cc:343] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.307116 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 36f966bf512d4cbe9ae406386db31533 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fe2b75a3e59437c885abc18614f1da2) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.307567 849 leader_election.cc:304] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.308578 2095 raft_consensus.cc:2802] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.308698 2095 raft_consensus.cc:491] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.309473 2095 raft_consensus.cc:3058] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.310787 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:29.311606 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:29.313133 2095 raft_consensus.cc:513] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.314005 2095 leader_election.cc:290] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:29.314412 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet f034b8cb5d8943938cb2b58c33b45494 consensus state has no leader
I20250905 19:52:29.315769 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb72fe6049894527917e1481bcb7b03a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.316035 1048 raft_consensus.cc:3058] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.317220 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:29.317647 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:29.319170 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:29.319218 1048 raft_consensus.cc:2466] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.319563 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:29.320325 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:29.321456 2108 raft_consensus.cc:491] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.322875 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.323166 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.323375 756 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.323804 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:29.324414 2407 tablet.cc:1722] T 05ec559d0f83445892f8285f9bdeb2c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:29.324462 2095 raft_consensus.cc:491] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.324736 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.327679 775 raft_consensus.cc:3058] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.330134 1169 tablet_bootstrap.cc:492] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
W20250905 19:52:29.330211 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 3315d3e9633e42ea8d7a6b5934cb0438 consensus state has no leader
I20250905 19:52:29.330400 1169 ts_tablet_manager.cc:1397] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.086s user 0.004s sys 0.000s
I20250905 19:52:29.331492 1169 raft_consensus.cc:357] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.331769 1169 raft_consensus.cc:383] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.331897 1169 raft_consensus.cc:738] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.332369 1169 consensus_queue.cc:260] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.334232 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 0fb8fe93061848a0b055979bad10f08b consensus state has no leader
I20250905 19:52:29.334887 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.335490 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.335726 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fb72fe6049894527917e1481bcb7b03a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.336084 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.334637 1532 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 239539ac19124de192cadae494720a2b consensus state has no leader
W20250905 19:52:29.337740 1547 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet e44e88a0df444fe5b156a8fdf3199507 consensus state has no leader
I20250905 19:52:29.342195 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.342234 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.343292 1798 raft_consensus.cc:491] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.343394 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4204f61929498188c18529148a8d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.343431 1798 raft_consensus.cc:513] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.343935 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:29.344101 1798 leader_election.cc:290] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.345043 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.345203 741 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.345376 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 262bcd1cfc994bb58e999b141b9bc7ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.345398 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:29.345096 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.346019 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:29.346608 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:29.346720 1169 ts_tablet_manager.cc:1428] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20250905 19:52:29.346717 2095 raft_consensus.cc:513] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.347071 1169 tablet_bootstrap.cc:492] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.347304 2376 ts_tablet_manager.cc:1918] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.347470 2376 log.cc:1199] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:29.348510 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.349277 1169 tablet_bootstrap.cc:654] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.350323 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.350348 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:29.350521 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
W20250905 19:52:29.354298 1547 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 65) took 3178 ms (client timeout 1000 ms). Trace:
W20250905 19:52:29.354431 1547 rpcz_store.cc:269] 0905 19:52:26.176336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.288364 (+112028us) service_pool.cc:225] Handling call
0905 19:52:26.305644 (+ 17280us) spinlock_profiling.cc:243] Waited 5.45 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec16f8 00007fabe1cfbc86 00007fabe1cfb9b5 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:27.951924 (+1646280us) spinlock_profiling.cc:243] Waited 4.74 ms on lock 0x61b0002be8e8. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fabead2a438 00007fabead257f1 00007fabead20131 00007fabeaec3887 00007fabe1cfbc86 00007fabe1cfb9b5 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:29.354257 (+1402333us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:27.934036 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:27.934191 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:27.934199 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:27.934201 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:27.934319 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:27.934333 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:27.934339 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:27.934418 (+ 79us) tablet.cc:699] Row locks acquired
0905 19:52:27.934422 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:27.934474 (+ 52us) write_op.cc:270] Start()
0905 19:52:27.934523 (+ 49us) write_op.cc:276] Timestamp: P: 1757101947934446 usec, L: 0
0905 19:52:27.934530 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:27.934761 (+ 231us) log.cc:844] Serialized 356 byte log entry
0905 19:52:27.937755 (+ 2994us) op_driver.cc:464] REPLICATION: finished
0905 19:52:27.938116 (+ 361us) write_op.cc:301] APPLY: starting
0905 19:52:27.938138 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:27.938192 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:27.938236 (+ 44us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:27.938303 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:27.938347 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:27.938359 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:27.938445 (+ 86us) log.cc:844] Serialized 23 byte log entry
0905 19:52:27.938508 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:27.938528 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:27.938537 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:27.938614 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:27.938618 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":211850,"reported_tablets":198,"spinlock_wait_cycles":23458944,"child_traces":[["op",{"apply.queue_time_us":292,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":495,"mutex_wait_us":354,"num_ops":1,"prepare.queue_time_us":389,"prepare.run_cpu_time_us":902,"prepare.run_wall_time_us":1248,"raft.queue_time_us":459,"raft.run_cpu_time_us":275,"raft.run_wall_time_us":275,"replication_time_us":3191}]]}
I20250905 19:52:29.355780 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.356544 1532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 73) took 3008 ms (client timeout 1000 ms). Trace:
W20250905 19:52:29.356662 1532 rpcz_store.cc:269] 0905 19:52:26.348435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.356478 (+ 8043us) service_pool.cc:225] Handling call
0905 19:52:29.356499 (+3000021us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:27.170861 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:27.171009 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:27.171020 (+ 11us) write_op.cc:435] Acquired schema lock
0905 19:52:27.171024 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:27.171306 (+ 282us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:27.171324 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:27.171329 (+ 5us) tablet.cc:683] Acquiring locks for 8 operations
0905 19:52:27.171495 (+ 166us) tablet.cc:699] Row locks acquired
0905 19:52:27.171498 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:27.171539 (+ 41us) write_op.cc:270] Start()
0905 19:52:27.171577 (+ 38us) write_op.cc:276] Timestamp: P: 1757101947171512 usec, L: 0
0905 19:52:27.171581 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:27.171841 (+ 260us) log.cc:844] Serialized 3146 byte log entry
0905 19:52:27.226635 (+ 54794us) op_driver.cc:464] REPLICATION: finished
0905 19:52:27.262849 (+ 36214us) write_op.cc:301] APPLY: starting
0905 19:52:27.262883 (+ 34us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:27.262984 (+ 101us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:27.263018 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:27.263174 (+ 156us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:27.263215 (+ 41us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=8
0905 19:52:27.263222 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:27.263333 (+ 111us) log.cc:844] Serialized 65 byte log entry
0905 19:52:27.263907 (+ 574us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:27.263938 (+ 31us) write_op.cc:454] Released schema lock
0905 19:52:27.263947 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:27.264042 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:27.264047 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":247895,"reported_tablets":178,"child_traces":[["op",{"apply.queue_time_us":36124,"apply.run_cpu_time_us":715,"apply.run_wall_time_us":1196,"num_ops":8,"prepare.queue_time_us":22869,"prepare.run_cpu_time_us":1106,"prepare.run_wall_time_us":1105,"replication_time_us":55016,"thread_start_us":60958,"threads_started":2}]]}
I20250905 19:52:29.358000 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.358203 1547 master_service.cc:948] ReplaceTablet: received request to replace tablet d296428b70f9411190b6c408d81639f2 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:29.358361 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.358368 849 leader_election.cc:304] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:29.358623 848 leader_election.cc:343] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.358659 986 leader_election.cc:343] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.359428 2376 ts_tablet_manager.cc:1939] T 8ed6109bdf8f4caf8c223a42401b94ab P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:29.359488 1584 raft_consensus.cc:2802] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.359609 1584 raft_consensus.cc:695] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:29.359864 1584 consensus_queue.cc:237] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.360743 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.359611 881 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.360960 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.361661 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_e02928704120437bbdda2c9819761a85"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:29.362263 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e02928704120437bbdda2c9819761a85 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.363933 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.365960 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.367868 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.368180 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f581403ec924fb0948400df41ed8aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.369388 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.369781 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.370457 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.371229 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.373286 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.373412 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.373489 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.373571 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.373888 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.375564 752 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.375841 984 leader_election.cc:343] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.376041 984 leader_election.cc:304] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.376232 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.376412 756 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.376446 1798 raft_consensus.cc:2747] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.376536 2100 raft_consensus.cc:491] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.376693 2100 raft_consensus.cc:513] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.376935 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 38) took 2878 ms. Trace:
I20250905 19:52:29.377063 1532 rpcz_store.cc:276] 0905 19:52:26.497972 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.359276 (+2861304us) service_pool.cc:225] Handling call
0905 19:52:29.362357 (+ 3081us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.362791 (+ 434us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.373348 (+ 10557us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.373505 (+ 157us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.376885 (+ 3380us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.363351 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.363496 (+ 145us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.363506 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:29.363510 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:29.363637 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.363656 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.363664 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.363768 (+ 104us) tablet.cc:699] Row locks acquired
0905 19:52:29.363773 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:29.363812 (+ 39us) write_op.cc:270] Start()
0905 19:52:29.363853 (+ 41us) write_op.cc:276] Timestamp: P: 1757101949363792 usec, L: 0
0905 19:52:29.363858 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.364090 (+ 232us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.372431 (+ 8341us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.372599 (+ 168us) write_op.cc:301] APPLY: starting
0905 19:52:29.372623 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.372730 (+ 107us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.372766 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.372942 (+ 176us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.372996 (+ 54us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.373009 (+ 13us) write_op.cc:312] APPLY: finished
0905 19:52:29.373117 (+ 108us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.373217 (+ 100us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.373249 (+ 32us) write_op.cc:454] Released schema lock
0905 19:52:29.373260 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.373341 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.373345 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":102,"apply.run_cpu_time_us":738,"apply.run_wall_time_us":738,"num_ops":2,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":886,"prepare.run_wall_time_us":886,"raft.queue_time_us":904,"raft.run_cpu_time_us":280,"raft.run_wall_time_us":281,"replication_time_us":8528}]]}
I20250905 19:52:29.377590 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.377749 1849 raft_consensus.cc:491] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.378340 2100 leader_election.cc:290] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.378410 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_3ab8cb26205842d09aac7a486a568cd8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:29.378615 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.379238 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3ab8cb26205842d09aac7a486a568cd8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.379427 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.379843 747 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.379985 1849 raft_consensus.cc:513] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.381191 1849 leader_election.cc:290] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.381219 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.381539 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.381695 737 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.381844 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.381966 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.382099 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.382231 741 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.382360 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.382486 749 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.382743 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.383586 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.383945 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.384348 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.384681 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.385279 1798 raft_consensus.cc:491] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.385331 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.385414 1798 raft_consensus.cc:513] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.385715 749 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.386059 1798 leader_election.cc:290] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.386694 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.387087 984 leader_election.cc:343] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.392316 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.393090 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.393154 711 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:29.393697 748 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.393769 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 39) took 2887 ms. Trace:
I20250905 19:52:29.394102 2379 ts_tablet_manager.cc:1918] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.393900 1532 rpcz_store.cc:276] 0905 19:52:26.506681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.378207 (+2871526us) service_pool.cc:225] Handling call
0905 19:52:29.379342 (+ 1135us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.380043 (+ 701us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.393511 (+ 13468us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.393618 (+ 107us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.393720 (+ 102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.383371 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.383503 (+ 132us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.383510 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:29.383514 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:29.383630 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.383645 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.383653 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.383739 (+ 86us) tablet.cc:699] Row locks acquired
0905 19:52:29.383743 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:29.383788 (+ 45us) write_op.cc:270] Start()
0905 19:52:29.383829 (+ 41us) write_op.cc:276] Timestamp: P: 1757101949383766 usec, L: 0
0905 19:52:29.383833 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.384095 (+ 262us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.392746 (+ 8651us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.392942 (+ 196us) write_op.cc:301] APPLY: starting
0905 19:52:29.392961 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.393008 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.393046 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.393161 (+ 115us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.393203 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.393210 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:29.393300 (+ 90us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.393386 (+ 86us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.393416 (+ 30us) write_op.cc:454] Released schema lock
0905 19:52:29.393427 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.393510 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.393513 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":134,"apply.run_cpu_time_us":563,"apply.run_wall_time_us":562,"num_ops":2,"prepare.queue_time_us":2822,"prepare.run_cpu_time_us":869,"prepare.run_wall_time_us":869,"raft.queue_time_us":1340,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":321,"replication_time_us":8850}]]}
I20250905 19:52:29.394342 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.394755 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_56ae03b8d3eb4a58970a492506fcc6f1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:29.394912 2379 log.cc:1199] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/74a7e7d9874e4d26bda3ab24a6e24adc
W20250905 19:52:29.395385 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_56ae03b8d3eb4a58970a492506fcc6f1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.395407 748 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.396239 2379 ts_tablet_manager.cc:1939] T 74a7e7d9874e4d26bda3ab24a6e24adc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:29.396600 741 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.396986 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.398051 2379 tablet_replica.cc:331] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:29.398312 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.398655 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.399036 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 813f208881ac4782874138987a710d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.399407 739 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.399885 740 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.400679 2379 raft_consensus.cc:2241] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.401000 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:29.402107 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6bb79e3126074736a23b128a8e14fab6 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6bb79e3126074736a23b128a8e14fab6 already in progress: creating tablet
W20250905 19:52:29.402453 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2350bd08f977424ba94ee8fa01f66efa because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 2350bd08f977424ba94ee8fa01f66efa already in progress: creating tablet
W20250905 19:52:29.402673 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
W20250905 19:52:29.402956 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 1f1a8e7059b348cba34069ef49bf1aad already in progress: creating tablet
I20250905 19:52:29.403025 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:29.403154 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 74a7e7d9874e4d26bda3ab24a6e24adc (table default.Table_aebe460a114e490ab8dfed41377a96b8 [id=f03c03ae24744485b3ac71752d70f7d8]) successfully deleted
I20250905 19:52:29.403251 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:29.403558 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:29.403715 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:29.403884 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:29.404150 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:29.404690 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 40) took 2897 ms. Trace:
I20250905 19:52:29.404816 1532 rpcz_store.cc:276] 0905 19:52:26.506810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.394541 (+2887731us) service_pool.cc:225] Handling call
0905 19:52:29.395482 (+ 941us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.395906 (+ 424us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.404458 (+ 8552us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.404556 (+ 98us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.404647 (+ 91us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.396922 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.397056 (+ 134us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.397062 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:29.397064 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:29.397157 (+ 93us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.397171 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.397177 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.397249 (+ 72us) tablet.cc:699] Row locks acquired
0905 19:52:29.397251 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:29.397290 (+ 39us) write_op.cc:270] Start()
0905 19:52:29.397321 (+ 31us) write_op.cc:276] Timestamp: P: 1757101949397269 usec, L: 0
0905 19:52:29.397324 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.397493 (+ 169us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.403777 (+ 6284us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.403936 (+ 159us) write_op.cc:301] APPLY: starting
0905 19:52:29.403952 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.404017 (+ 65us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.404051 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.404173 (+ 122us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.404209 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.404217 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:29.404298 (+ 81us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.404357 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.404379 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:29.404386 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.404461 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.404466 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":106,"apply.run_cpu_time_us":520,"apply.run_wall_time_us":519,"num_ops":2,"prepare.queue_time_us":531,"prepare.run_cpu_time_us":680,"prepare.run_wall_time_us":680,"raft.queue_time_us":1053,"raft.run_cpu_time_us":272,"raft.run_wall_time_us":273,"replication_time_us":6421}]]}
I20250905 19:52:29.405651 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_57684ee2f11b44129e0b3d35029151fc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:29.406322 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_57684ee2f11b44129e0b3d35029151fc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.406358 2379 raft_consensus.cc:2270] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.407070 2379 ts_tablet_manager.cc:1905] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.408478 1849 raft_consensus.cc:491] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.408640 1849 raft_consensus.cc:513] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.409421 1849 leader_election.cc:290] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:29.410458 711 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0d4204f61929498188c18529148a8d13
I20250905 19:52:29.410930 1849 raft_consensus.cc:491] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.411072 1849 raft_consensus.cc:513] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.411713 1849 leader_election.cc:290] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.412089 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:29.412586 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
W20250905 19:52:29.412647 711 leader_election.cc:343] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9
I20250905 19:52:29.412808 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:29.412997 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:29.413184 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:29.414283 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:29.414350 2376 tablet_replica.cc:331] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:29.414872 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:29.415258 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:29.415452 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:29.415838 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:29.416206 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:29.422377 1798 raft_consensus.cc:491] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.422559 1798 raft_consensus.cc:513] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.423321 1798 leader_election.cc:290] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.422101 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 41) took 2887 ms. Trace:
I20250905 19:52:29.423888 1532 rpcz_store.cc:276] 0905 19:52:26.535078 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.405444 (+2870366us) service_pool.cc:225] Handling call
0905 19:52:29.406435 (+ 991us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.406973 (+ 538us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.421831 (+ 14858us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.421942 (+ 111us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.422053 (+ 111us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.408283 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.408434 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.408443 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:29.408448 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:29.408573 (+ 125us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.408587 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.408592 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.408670 (+ 78us) tablet.cc:699] Row locks acquired
0905 19:52:29.408675 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:29.408737 (+ 62us) write_op.cc:270] Start()
0905 19:52:29.408776 (+ 39us) write_op.cc:276] Timestamp: P: 1757101949408721 usec, L: 0
0905 19:52:29.408782 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.408998 (+ 216us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.421172 (+ 12174us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.421309 (+ 137us) write_op.cc:301] APPLY: starting
0905 19:52:29.421324 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.421372 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.421398 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.421512 (+ 114us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.421548 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.421555 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:29.421634 (+ 79us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.421702 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.421736 (+ 34us) write_op.cc:454] Released schema lock
0905 19:52:29.421743 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.421819 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.421823 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":502,"apply.run_wall_time_us":502,"num_ops":2,"prepare.queue_time_us":751,"prepare.run_cpu_time_us":823,"prepare.run_wall_time_us":823,"raft.queue_time_us":101,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":12356}]]}
I20250905 19:52:29.425045 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_e34bbf7d7da54ba3be25ddf22622b103"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:29.425793 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e34bbf7d7da54ba3be25ddf22622b103 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.429764 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:29.430228 1540 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
W20250905 19:52:29.432090 711 consensus_peers.cc:489] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.432269 2095 leader_election.cc:290] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.432988 2376 raft_consensus.cc:2241] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:29.433332 2376 raft_consensus.cc:2270] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.434425 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 42) took 2872 ms. Trace:
I20250905 19:52:29.434558 1532 rpcz_store.cc:276] 0905 19:52:26.561925 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.424791 (+2862866us) service_pool.cc:225] Handling call
0905 19:52:29.425927 (+ 1136us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.426497 (+ 570us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.434175 (+ 7678us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.434280 (+ 105us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.434375 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.428256 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.428406 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.428415 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:29.428420 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:29.428547 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.428566 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.428573 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.428674 (+ 101us) tablet.cc:699] Row locks acquired
0905 19:52:29.428678 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:29.428742 (+ 64us) write_op.cc:270] Start()
0905 19:52:29.428807 (+ 65us) write_op.cc:276] Timestamp: P: 1757101949428723 usec, L: 0
0905 19:52:29.428814 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.429050 (+ 236us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.433435 (+ 4385us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.433556 (+ 121us) write_op.cc:301] APPLY: starting
0905 19:52:29.433572 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.433622 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.433680 (+ 58us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.433847 (+ 167us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.433890 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.433900 (+ 10us) write_op.cc:312] APPLY: finished
0905 19:52:29.433993 (+ 93us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.434064 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.434093 (+ 29us) write_op.cc:454] Released schema lock
0905 19:52:29.434101 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.434176 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.434181 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":75,"apply.run_cpu_time_us":611,"apply.run_wall_time_us":610,"num_ops":2,"prepare.queue_time_us":1137,"prepare.run_cpu_time_us":917,"prepare.run_wall_time_us":917,"raft.queue_time_us":988,"raft.run_cpu_time_us":295,"raft.run_wall_time_us":294,"replication_time_us":4585}]]}
I20250905 19:52:29.435304 1532 master_service.cc:948] ReplaceTablet: received request to replace tablet 19e4edddafcd4d22a0e0ae66b0b953e0 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:29.435652 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.436674 1532 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 43) took 2839 ms. Trace:
I20250905 19:52:29.436774 1532 rpcz_store.cc:276] 0905 19:52:26.597363 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.435245 (+2837882us) service_pool.cc:225] Handling call
0905 19:52:29.436619 (+ 1374us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:29.437754 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_44734742f0c948af9cce54466ff759c6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:29.438405 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_44734742f0c948af9cce54466ff759c6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.438433 2379 ts_tablet_manager.cc:1918] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.438473 891 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.438565 2379 log.cc:1199] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:29.438959 2379 ts_tablet_manager.cc:1939] T 4c76077e4adf45a7adc736b0e69032eb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:29.439563 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4c76077e4adf45a7adc736b0e69032eb (table default.Table_48447ad862474825a10962298c97650d [id=d467b5191bf744858924c0ecbbf2e53c]) successfully deleted
I20250905 19:52:29.439666 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239539ac19124de192cadae494720a2b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.440120 2376 ts_tablet_manager.cc:1905] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.445259 1798 raft_consensus.cc:491] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.445444 1798 raft_consensus.cc:513] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.446173 1798 leader_election.cc:290] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:29.446588 984 leader_election.cc:343] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.446784 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.447801 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.448501 891 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.449753 891 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.450201 2412 raft_consensus.cc:491] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.450356 2412 raft_consensus.cc:513] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.451119 2412 leader_election.cc:290] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.451539 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.453141 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 813f208881ac4782874138987a710d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.454161 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.454361 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.454447 882 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.454557 893 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.454739 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.455065 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.455276 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.455332 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8ed6109bdf8f4caf8c223a42401b94ab (table default.Table_68f98d1c0de04ff9a6ad5188ab275ec9 [id=711f36b2be7c4a368fcb3106e691b133]) successfully deleted
I20250905 19:52:29.455456 877 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.455603 883 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:29.455689 984 leader_election.cc:343] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.455770 889 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.457028 770 raft_consensus.cc:3058] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.457448 2413 raft_consensus.cc:491] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.457568 2413 raft_consensus.cc:513] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.458251 2413 leader_election.cc:290] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.463287 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 44) took 2798 ms. Trace:
I20250905 19:52:29.463421 1532 rpcz_store.cc:276] 0905 19:52:26.665011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.437536 (+2772525us) service_pool.cc:225] Handling call
0905 19:52:29.438502 (+ 966us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.457054 (+ 18552us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.463009 (+ 5955us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.463119 (+ 110us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.463214 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.457700 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.457847 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.457856 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:29.457860 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:29.458005 (+ 145us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.458026 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.458033 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.458113 (+ 80us) tablet.cc:699] Row locks acquired
0905 19:52:29.458116 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:29.458150 (+ 34us) write_op.cc:270] Start()
0905 19:52:29.458182 (+ 32us) write_op.cc:276] Timestamp: P: 1757101949458134 usec, L: 0
0905 19:52:29.458186 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.458358 (+ 172us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.462294 (+ 3936us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.462430 (+ 136us) write_op.cc:301] APPLY: starting
0905 19:52:29.462448 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.462521 (+ 73us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.462558 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.462696 (+ 138us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.462747 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.462756 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:29.462841 (+ 85us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.462907 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.462929 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:29.462935 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.463009 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.463014 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":84,"apply.run_cpu_time_us":572,"apply.run_wall_time_us":572,"num_ops":2,"prepare.queue_time_us":58,"prepare.run_cpu_time_us":773,"prepare.run_wall_time_us":774,"raft.queue_time_us":1077,"raft.run_cpu_time_us":250,"raft.run_wall_time_us":252,"replication_time_us":4074}]]}
I20250905 19:52:29.464761 1532 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_94f58de97c5946149d23d635eda4d555"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:29.465363 1532 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_94f58de97c5946149d23d635eda4d555 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:29.466042 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3609de83732d4ad8857372acfbb5c32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3609de83732d4ad8857372acfbb5c32a already in progress: creating tablet
I20250905 19:52:29.467056 2379 tablet_replica.cc:331] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:29.467322 2379 raft_consensus.cc:2241] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.467509 2379 raft_consensus.cc:2270] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.468178 2379 ts_tablet_manager.cc:1905] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:29.470517 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 111550us
I20250905 19:52:29.471939 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.472186 1532 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 45) took 2807 ms. Trace:
I20250905 19:52:29.443053 2406 tablet.cc:1722] T 05ec559d0f83445892f8285f9bdeb2c8 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:29.472306 1532 rpcz_store.cc:276] 0905 19:52:26.665147 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.464533 (+2799386us) service_pool.cc:225] Handling call
0905 19:52:29.465471 (+ 938us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.465972 (+ 501us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.471830 (+ 5858us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.472017 (+ 187us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.472125 (+ 108us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.466605 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.466714 (+ 109us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.466720 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:29.466722 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:29.466806 (+ 84us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.466819 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.466823 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.466890 (+ 67us) tablet.cc:699] Row locks acquired
0905 19:52:29.466893 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:29.466925 (+ 32us) write_op.cc:270] Start()
0905 19:52:29.466966 (+ 41us) write_op.cc:276] Timestamp: P: 1757101949466910 usec, L: 0
0905 19:52:29.466970 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.467124 (+ 154us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.470603 (+ 3479us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.470784 (+ 181us) write_op.cc:301] APPLY: starting
0905 19:52:29.470801 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.470849 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.470875 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.470987 (+ 112us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.471021 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.471029 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:29.471112 (+ 83us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.471172 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.471193 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:29.471199 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.471264 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.471268 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"apply.run_cpu_time_us":474,"apply.run_wall_time_us":474,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":628,"prepare.run_wall_time_us":628,"raft.queue_time_us":915,"raft.run_cpu_time_us":219,"raft.run_wall_time_us":218,"replication_time_us":3605}]]}
I20250905 19:52:29.477917 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.478250 775 raft_consensus.cc:2466] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.483702 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.488198 848 leader_election.cc:343] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.488560 848 leader_election.cc:304] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:29.488874 2413 raft_consensus.cc:2802] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.488972 2413 raft_consensus.cc:491] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.489055 2413 raft_consensus.cc:3058] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.489081 2386 raft_consensus.cc:491] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.489213 2386 raft_consensus.cc:513] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.490005 2386 leader_election.cc:290] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.490120 2386 raft_consensus.cc:491] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.490204 2386 raft_consensus.cc:513] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.490648 2386 leader_election.cc:290] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.491447 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2d929ec75d34cddad5286d8e37fcefb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.491680 2413 raft_consensus.cc:513] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.492002 984 leader_election.cc:343] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.492870 2413 leader_election.cc:290] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.494344 1547 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 37) took 3070 ms. Trace:
I20250905 19:52:29.494470 1547 rpcz_store.cc:276] 0905 19:52:26.423634 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.358130 (+2934496us) service_pool.cc:225] Handling call
0905 19:52:29.494261 (+136131us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:29.494695 1171 tablet_bootstrap.cc:492] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.494863 1171 ts_tablet_manager.cc:1397] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.303s user 0.009s sys 0.024s
I20250905 19:52:29.494977 1547 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_96c633b175ac41309951506efe002443"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:29.495633 1547 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_96c633b175ac41309951506efe002443 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:29.495666 1171 raft_consensus.cc:357] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.495862 1171 raft_consensus.cc:383] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250905 19:52:29.496655 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 137180us
I20250905 19:52:29.497506 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.498116 984 leader_election.cc:343] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.505164 1849 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.505355 1849 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.506305 1849 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.495937 1171 raft_consensus.cc:738] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.507359 1171 consensus_queue.cc:260] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.507704 848 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.507822 711 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:29.507905 1171 ts_tablet_manager.cc:1428] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:29.511544 1532 master_service.cc:948] ReplaceTablet: received request to replace tablet 9193487b4ab74dafa76a386ddcaf674b from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:29.515139 2407 tablet.cc:1722] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:29.516022 1547 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 46) took 2849 ms. Trace:
I20250905 19:52:29.516163 1547 rpcz_store.cc:276] 0905 19:52:26.666710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.494700 (+2827990us) service_pool.cc:225] Handling call
0905 19:52:29.495760 (+ 1060us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:29.509311 (+ 13551us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:29.515699 (+ 6388us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:29.515797 (+ 98us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:29.515894 (+ 97us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.509941 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.510169 (+ 228us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.510176 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:29.510179 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:29.510301 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.510316 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.510321 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.510401 (+ 80us) tablet.cc:699] Row locks acquired
0905 19:52:29.510403 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:29.510445 (+ 42us) write_op.cc:270] Start()
0905 19:52:29.510480 (+ 35us) write_op.cc:276] Timestamp: P: 1757101949510426 usec, L: 0
0905 19:52:29.510483 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.510692 (+ 209us) log.cc:844] Serialized 433 byte log entry
0905 19:52:29.515055 (+ 4363us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.515186 (+ 131us) write_op.cc:301] APPLY: starting
0905 19:52:29.515203 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.515252 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.515287 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.515404 (+ 117us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.515437 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:29.515446 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:29.515531 (+ 85us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.515595 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.515615 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:29.515622 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.515704 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.515708 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":80,"apply.run_cpu_time_us":510,"apply.run_wall_time_us":509,"num_ops":2,"prepare.queue_time_us":89,"prepare.run_cpu_time_us":881,"prepare.run_wall_time_us":882,"raft.queue_time_us":1110,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":266,"replication_time_us":4541}]]}
I20250905 19:52:29.516891 1547 master_service.cc:948] ReplaceTablet: received request to replace tablet 6221d7d058944bdb9aa730791cf2417b from {username='slave'} at 127.0.0.1:56416
W20250905 19:52:29.530392 711 consensus_peers.cc:489] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.533982 1849 raft_consensus.cc:491] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.534142 1849 raft_consensus.cc:513] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.534727 1849 leader_election.cc:290] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:29.536033 711 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
I20250905 19:52:29.544526 737 tablet_service.cc:1468] Processing CreateTablet for tablet e3cb1b098f42441594e28aeb1a9dbbf1 (DEFAULT_TABLE table=default.Table_e02928704120437bbdda2c9819761a85 [id=c08a19b2c2384f52b2f91b47ef74e13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:29.544973 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cb1b098f42441594e28aeb1a9dbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:29.544912 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.545123 1046 raft_consensus.cc:2466] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.545106 1025 tablet_service.cc:1468] Processing CreateTablet for tablet e3cb1b098f42441594e28aeb1a9dbbf1 (DEFAULT_TABLE table=default.Table_e02928704120437bbdda2c9819761a85 [id=c08a19b2c2384f52b2f91b47ef74e13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:29.545534 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cb1b098f42441594e28aeb1a9dbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:29.545805 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.545949 1046 raft_consensus.cc:2466] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.549849 2407 tablet.cc:1722] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:29.554378 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:29.555568 1849 raft_consensus.cc:491] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.555703 1849 raft_consensus.cc:513] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.557174 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "42f3c1efb94e446fb0128b6b412821e2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.557374 775 raft_consensus.cc:3058] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.561273 775 raft_consensus.cc:2466] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.561961 1849 leader_election.cc:290] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.562765 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.570334 1547 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 6221d7d058944bdb9aa730791cf2417b
I20250905 19:52:29.571316 1547 catalog_manager.cc:6652] ReplaceTablet: tablet 6221d7d058944bdb9aa730791cf2417b deleted and replaced by tablet 9f5d19f0216a4f069237e48c62ee7875
I20250905 19:52:29.571458 1547 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 48) took 2504 ms. Trace:
I20250905 19:52:29.571563 1547 rpcz_store.cc:276] 0905 19:52:27.066836 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.516805 (+2449969us) service_pool.cc:225] Handling call
0905 19:52:29.571429 (+ 54624us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.540191 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.540375 (+ 184us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.540385 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:29.540388 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:29.540534 (+ 146us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.540553 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.540559 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.540655 (+ 96us) tablet.cc:699] Row locks acquired
0905 19:52:29.540659 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:29.540701 (+ 42us) write_op.cc:270] Start()
0905 19:52:29.540743 (+ 42us) write_op.cc:276] Timestamp: P: 1757101949540683 usec, L: 0
0905 19:52:29.540748 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.541025 (+ 277us) log.cc:844] Serialized 622 byte log entry
0905 19:52:29.557475 (+ 16450us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.557608 (+ 133us) write_op.cc:301] APPLY: starting
0905 19:52:29.557627 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.557678 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.557715 (+ 37us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.557820 (+ 105us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.557878 (+ 58us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:29.557886 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:29.557963 (+ 77us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.558026 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.558045 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:29.558062 (+ 17us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.558130 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.558134 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":85,"apply.run_cpu_time_us":514,"apply.run_wall_time_us":513,"num_ops":2,"prepare.queue_time_us":539,"prepare.run_cpu_time_us":969,"prepare.run_wall_time_us":969,"raft.queue_time_us":752,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"replication_time_us":16696}]]}
I20250905 19:52:29.574100 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 6221d7d058944bdb9aa730791cf2417b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d19f0216a4f069237e48c62ee7875) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.574327 2435 tablet.cc:1722] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:29.576521 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 6221d7d058944bdb9aa730791cf2417b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d19f0216a4f069237e48c62ee7875) from {username='slave'} at 127.0.0.1:39172
W20250905 19:52:29.577685 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6221d7d058944bdb9aa730791cf2417b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6221d7d058944bdb9aa730791cf2417b already in progress: creating tablet
I20250905 19:52:29.579792 2435 tablet.cc:1722] T 05ec559d0f83445892f8285f9bdeb2c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:29.583611 1170 tablet_bootstrap.cc:492] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.583771 1170 ts_tablet_manager.cc:1397] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.450s user 0.005s sys 0.023s
I20250905 19:52:29.584121 1532 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 9193487b4ab74dafa76a386ddcaf674b
I20250905 19:52:29.584581 1170 raft_consensus.cc:357] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.584745 1170 raft_consensus.cc:383] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.584800 1170 raft_consensus.cc:738] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.585978 2407 tablet.cc:1722] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.586227 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 9193487b4ab74dafa76a386ddcaf674b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8bbbcac06944469e13f14ab1d80402) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:29.586372 1169 tablet_bootstrap.cc:492] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.586349 2431 tablet.cc:1722] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:29.586494 1169 ts_tablet_manager.cc:1397] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.240s user 0.002s sys 0.002s
W20250905 19:52:29.586961 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 9193487b4ab74dafa76a386ddcaf674b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9193487b4ab74dafa76a386ddcaf674b already in progress: creating tablet
I20250905 19:52:29.587293 1169 raft_consensus.cc:357] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.587528 1169 raft_consensus.cc:383] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.587606 1169 raft_consensus.cc:738] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.587839 1532 catalog_manager.cc:6652] ReplaceTablet: tablet 9193487b4ab74dafa76a386ddcaf674b deleted and replaced by tablet 6b8bbbcac06944469e13f14ab1d80402
I20250905 19:52:29.587841 1169 consensus_queue.cc:260] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.588400 1169 ts_tablet_manager.cc:1428] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:29.588682 1169 tablet_bootstrap.cc:492] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.589960 1532 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 47) took 2605 ms. Trace:
I20250905 19:52:29.590407 1169 tablet_bootstrap.cc:654] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.594609 1532 rpcz_store.cc:276] 0905 19:52:26.984205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.511433 (+2527228us) service_pool.cc:225] Handling call
0905 19:52:29.589924 (+ 78491us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.553643 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.553793 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.553801 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:29.553804 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:29.553931 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.553944 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.553949 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:29.554027 (+ 78us) tablet.cc:699] Row locks acquired
0905 19:52:29.554029 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:29.554075 (+ 46us) write_op.cc:270] Start()
0905 19:52:29.554110 (+ 35us) write_op.cc:276] Timestamp: P: 1757101949554057 usec, L: 0
0905 19:52:29.554114 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.554305 (+ 191us) log.cc:844] Serialized 622 byte log entry
0905 19:52:29.561682 (+ 7377us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.561848 (+ 166us) write_op.cc:301] APPLY: starting
0905 19:52:29.561866 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.561920 (+ 54us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.561965 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.562089 (+ 124us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.562168 (+ 79us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:29.562178 (+ 10us) write_op.cc:312] APPLY: finished
0905 19:52:29.562266 (+ 88us) log.cc:844] Serialized 29 byte log entry
0905 19:52:29.562342 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.562368 (+ 26us) write_op.cc:454] Released schema lock
0905 19:52:29.562377 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:29.562459 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.562465 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":112,"apply.run_cpu_time_us":599,"apply.run_wall_time_us":598,"num_ops":2,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":759,"prepare.run_wall_time_us":760,"raft.queue_time_us":165,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":7533}]]}
I20250905 19:52:29.596294 2379 ts_tablet_manager.cc:1918] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.596321 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9193487b4ab74dafa76a386ddcaf674b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8bbbcac06944469e13f14ab1d80402) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:29.596405 2379 log.cc:1199] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:29.596807 2379 ts_tablet_manager.cc:1939] T 4fe94d1254d04949aa25346d8e323c3b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:29.540845 2376 ts_tablet_manager.cc:1918] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:29.600124 2376 log.cc:1199] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/2683a36998674f739b41d2a57339954c
I20250905 19:52:29.600224 1171 tablet_bootstrap.cc:492] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.582971 2007 raft_consensus.cc:491] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.600396 2386 raft_consensus.cc:491] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.600548 2386 raft_consensus.cc:513] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.600579 2007 raft_consensus.cc:513] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.585305 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "89a73e984cbb4268b563f1ca01f4c65d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.601238 2007 leader_election.cc:290] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.601222 2386 leader_election.cc:290] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.593881 2415 raft_consensus.cc:491] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.601392 2415 raft_consensus.cc:513] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.601887 2415 leader_election.cc:290] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.604121 1170 consensus_queue.cc:260] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.605533 1170 ts_tablet_manager.cc:1428] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.022s user 0.002s sys 0.000s
I20250905 19:52:29.613044 2406 tablet.cc:1722] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:29.613490 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.614068 984 leader_election.cc:343] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.617249 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4fe94d1254d04949aa25346d8e323c3b (table default.Table_32577b5aebfd4f37b4400d62e431d7f1 [id=e0eba87919924759a4a5beae73fd3ac9]) successfully deleted
W20250905 19:52:29.622449 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 578a6b9224cc4d8585d6c4b241762e1b already in progress: creating tablet
I20250905 19:52:29.623701 1170 tablet_bootstrap.cc:492] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.626011 1170 tablet_bootstrap.cc:654] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.633471 2379 tablet_replica.cc:331] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:29.633430 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 37894e39e6f2417fbfcd6d43f1ea2b37 already in progress: creating tablet
I20250905 19:52:29.633808 2379 raft_consensus.cc:2241] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:29.634101 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a2ff206366ad4344b4df5fc6a5584f8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ff206366ad4344b4df5fc6a5584f8a already in progress: creating tablet
W20250905 19:52:29.634670 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a1b4989f17454aa58c05727b797c2009 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a1b4989f17454aa58c05727b797c2009 already in progress: creating tablet
I20250905 19:52:29.634730 2379 raft_consensus.cc:2270] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:29.635167 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd1280fefc3e43db9928567a734644f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1280fefc3e43db9928567a734644f1 already in progress: creating tablet
I20250905 19:52:29.635771 2379 ts_tablet_manager.cc:1905] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:29.635881 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e because tablet deleting was already in progress. No further retry: Already present: State transition of tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e already in progress: creating tablet
I20250905 19:52:29.643918 1849 raft_consensus.cc:491] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.644189 1849 raft_consensus.cc:513] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.645015 1849 leader_election.cc:290] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:29.646227 711 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2f581403ec924fb0948400df41ed8aab
I20250905 19:52:29.648182 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 6221d7d058944bdb9aa730791cf2417b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d19f0216a4f069237e48c62ee7875) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.648319 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 9193487b4ab74dafa76a386ddcaf674b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8bbbcac06944469e13f14ab1d80402) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:29.649559 2376 ts_tablet_manager.cc:1939] T 2683a36998674f739b41d2a57339954c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:29.650188 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a518797d26140f5b9abe41eab415040" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.651547 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00ac56c449ba4b7b8dbd5e87120799d9" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.651732 913 raft_consensus.cc:2466] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:29.652943 1169 tablet_bootstrap.cc:492] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.653097 1169 ts_tablet_manager.cc:1397] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.015s
I20250905 19:52:29.653713 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "53ac46ce05694bc2b53e195638be9e23" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.648036 848 leader_election.cc:304] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:29.654498 2109 raft_consensus.cc:2802] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.654632 1169 raft_consensus.cc:357] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.654814 1169 raft_consensus.cc:383] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.654881 1169 raft_consensus.cc:738] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.655069 1169 consensus_queue.cc:260] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.655537 1169 ts_tablet_manager.cc:1428] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:29.655853 1169 tablet_bootstrap.cc:492] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.657878 1169 tablet_bootstrap.cc:654] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.659286 878 tablet_service.cc:1468] Processing CreateTablet for tablet e3cb1b098f42441594e28aeb1a9dbbf1 (DEFAULT_TABLE table=default.Table_e02928704120437bbdda2c9819761a85 [id=c08a19b2c2384f52b2f91b47ef74e13f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:29.659722 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e3cb1b098f42441594e28aeb1a9dbbf1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:29.671721 711 consensus_peers.cc:489] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.680852 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 2683a36998674f739b41d2a57339954c (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]) successfully deleted
I20250905 19:52:29.681430 2376 tablet_replica.cc:331] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:29.681429 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 3609de83732d4ad8857372acfbb5c32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3609de83732d4ad8857372acfbb5c32a already in progress: creating tablet
I20250905 19:52:29.681787 2376 raft_consensus.cc:2241] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:29.682205 2376 raft_consensus.cc:2270] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.683534 2376 ts_tablet_manager.cc:1905] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.687541 1170 tablet_bootstrap.cc:492] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.687687 1170 ts_tablet_manager.cc:1397] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.000s
I20250905 19:52:29.688483 1170 raft_consensus.cc:357] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.688656 1170 raft_consensus.cc:383] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.688732 1170 raft_consensus.cc:738] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:29.688588 2421 log.cc:927] Time spent T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.146s user 0.000s sys 0.000s
I20250905 19:52:29.688977 2109 raft_consensus.cc:695] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:29.689338 2109 consensus_queue.cc:237] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.690603 2379 ts_tablet_manager.cc:1918] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.699765 2406 tablet.cc:1722] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.705498 2428 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.692508 1170 consensus_queue.cc:260] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.693382 712 consensus_peers.cc:489] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.706899 1849 consensus_queue.cc:1035] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.708793 2445 tablet.cc:1722] T 6e89d23c4cb64f568b25f92a278b9d5e P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.715297 1170 ts_tablet_manager.cc:1428] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.028s user 0.003s sys 0.000s
I20250905 19:52:29.717587 2406 tablet.cc:1722] T 61667a379974487195f81043683b9a24 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:29.718423 1170 tablet_bootstrap.cc:492] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.720427 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc95a83ca872495894ef439cdda4e364" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.720660 1047 raft_consensus.cc:2466] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.720898 1170 tablet_bootstrap.cc:654] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.721419 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.721623 1053 raft_consensus.cc:2466] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.733290 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.733587 2435 tablet.cc:1722] T 6e89d23c4cb64f568b25f92a278b9d5e P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.734452 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc95a83ca872495894ef439cdda4e364" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.736454 770 raft_consensus.cc:3058] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.738497 2435 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.740965 2432 tablet.cc:1722] T 61667a379974487195f81043683b9a24 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:29.743656 712 leader_election.cc:304] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:29.744089 712 leader_election.cc:343] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:29.744210 712 leader_election.cc:304] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.744515 712 leader_election.cc:304] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.744774 2160 raft_consensus.cc:2747] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.744920 2160 raft_consensus.cc:2747] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.745029 2160 raft_consensus.cc:2747] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:29.745546 712 leader_election.cc:343] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 0d4204f61929498188c18529148a8d13
I20250905 19:52:29.745642 712 leader_election.cc:304] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.745939 712 leader_election.cc:304] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:29.745916 2160 consensus_queue.cc:1035] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.746143 1171 tablet_bootstrap.cc:654] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.746295 2160 raft_consensus.cc:2747] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.746454 2160 raft_consensus.cc:2802] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.746505 2160 raft_consensus.cc:491] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.746524 1169 tablet_bootstrap.cc:492] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:29.746562 2160 raft_consensus.cc:3058] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.746661 1169 ts_tablet_manager.cc:1397] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.091s user 0.017s sys 0.028s
I20250905 19:52:29.747020 2379 log.cc:1199] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:29.747327 2379 ts_tablet_manager.cc:1939] T 19e4edddafcd4d22a0e0ae66b0b953e0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:29.747936 2379 tablet_replica.cc:331] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:29.748159 2379 raft_consensus.cc:2241] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.748131 2443 raft_consensus.cc:491] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.748297 2379 raft_consensus.cc:2270] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.748275 2443 raft_consensus.cc:513] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.748940 2443 leader_election.cc:290] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.749191 2379 ts_tablet_manager.cc:1905] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.749305 2160 raft_consensus.cc:513] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.749843 2160 leader_election.cc:290] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.750900 2422 raft_consensus.cc:491] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.751339 2422 raft_consensus.cc:513] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.753005 2422 leader_election.cc:290] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.753381 1170 tablet_bootstrap.cc:492] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.753525 1170 ts_tablet_manager.cc:1397] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.000s
I20250905 19:52:29.754237 1170 raft_consensus.cc:357] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.754410 1170 raft_consensus.cc:383] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.754460 1170 raft_consensus.cc:738] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.754643 1170 consensus_queue.cc:260] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.754798 2422 raft_consensus.cc:491] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.754961 2422 raft_consensus.cc:513] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.755676 2422 leader_election.cc:290] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.756059 1170 ts_tablet_manager.cc:1428] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
W20250905 19:52:29.756872 711 leader_election.cc:343] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9
I20250905 19:52:29.758188 849 leader_election.cc:304] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.758522 2443 raft_consensus.cc:2802] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.758612 2443 raft_consensus.cc:491] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.758687 2443 raft_consensus.cc:3058] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.758791 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.758960 912 raft_consensus.cc:2466] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:29.759621 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239539ac19124de192cadae494720a2b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.759758 912 raft_consensus.cc:2466] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:29.759991 1169 raft_consensus.cc:357] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.760187 1169 raft_consensus.cc:383] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.760250 1169 raft_consensus.cc:738] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.760473 1169 consensus_queue.cc:260] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.760931 1169 ts_tablet_manager.cc:1428] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250905 19:52:29.761209 1169 tablet_bootstrap.cc:492] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.761448 774 raft_consensus.cc:3058] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.761402 2443 raft_consensus.cc:513] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.761936 2443 leader_election.cc:290] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.762876 1169 tablet_bootstrap.cc:654] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.762954 849 leader_election.cc:304] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.765492 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f52ff57fb1a4ea8a1ad4d5f0480c7f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.767692 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.768158 2443 raft_consensus.cc:2802] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.768244 2443 raft_consensus.cc:491] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.768324 2443 raft_consensus.cc:3058] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:29.746308 712 leader_election.cc:343] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
I20250905 19:52:29.770535 712 leader_election.cc:304] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.771127 2443 raft_consensus.cc:513] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.771327 1849 raft_consensus.cc:2747] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.771834 2443 leader_election.cc:290] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.772038 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.772267 712 leader_election.cc:343] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2f581403ec924fb0948400df41ed8aab
I20250905 19:52:29.772385 712 leader_election.cc:304] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:29.772676 849 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2
I20250905 19:52:29.772712 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.772889 1849 raft_consensus.cc:2747] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:29.773196 712 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
W20250905 19:52:29.773370 849 leader_election.cc:343] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2
I20250905 19:52:29.773640 712 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.774093 712 leader_election.cc:304] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.774322 849 leader_election.cc:304] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:29.774535 2444 raft_consensus.cc:2802] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.774595 2444 raft_consensus.cc:491] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.774648 2444 raft_consensus.cc:3058] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.774278 2464 raft_consensus.cc:2747] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.775863 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.776082 1170 tablet_bootstrap.cc:492] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.777351 2444 raft_consensus.cc:513] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.777871 1170 tablet_bootstrap.cc:654] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.777868 2444 leader_election.cc:290] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.778308 849 leader_election.cc:304] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:29.778923 2443 raft_consensus.cc:2802] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.779042 2443 raft_consensus.cc:491] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.779166 2443 raft_consensus.cc:3058] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.781459 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2d929ec75d34cddad5286d8e37fcefb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.782332 2443 raft_consensus.cc:513] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.782900 2443 leader_election.cc:290] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.782900 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.786172 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 19e4edddafcd4d22a0e0ae66b0b953e0 (table default.Table_ff5415e10957422498084d7730365989 [id=bd0b3b55bf3648b7a45b7aef904be570]) successfully deleted
I20250905 19:52:29.786556 1849 raft_consensus.cc:2747] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:29.789212 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8edf4b54974c427a855eee5792d75292 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8edf4b54974c427a855eee5792d75292 already in progress: creating tablet
I20250905 19:52:29.789805 1169 tablet_bootstrap.cc:492] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
W20250905 19:52:29.790561 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 52c99a76802b481c98f77dbd90de7cbb already in progress: creating tablet
I20250905 19:52:29.794209 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:29.800514 1849 raft_consensus.cc:491] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.800858 1849 raft_consensus.cc:513] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.801712 1849 leader_election.cc:290] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.806538 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00ac56c449ba4b7b8dbd5e87120799d9" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:29.806756 913 raft_consensus.cc:3058] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.809835 2422 consensus_queue.cc:1035] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.810757 913 raft_consensus.cc:2466] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:29.821884 2379 ts_tablet_manager.cc:1918] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.823791 1170 tablet_bootstrap.cc:492] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
W20250905 19:52:29.834625 711 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:29.842031 1170 ts_tablet_manager.cc:1397] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.066s user 0.006s sys 0.000s
I20250905 19:52:29.841406 2379 log.cc:1199] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a5af8232d31240ff855b79dcea29396b
I20250905 19:52:29.842775 2379 ts_tablet_manager.cc:1939] T a5af8232d31240ff855b79dcea29396b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:29.845865 1849 raft_consensus.cc:491] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.846007 1849 raft_consensus.cc:513] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.843618 1170 raft_consensus.cc:357] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.846480 1170 raft_consensus.cc:383] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.846560 1170 raft_consensus.cc:738] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.846609 1849 leader_election.cc:290] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.814038 2422 consensus_queue.cc:1035] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.848013 1170 consensus_queue.cc:260] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.848569 1170 ts_tablet_manager.cc:1428] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
W20250905 19:52:29.857607 711 consensus_peers.cc:489] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.823197 2428 tablet.cc:1722] T 4edfa9a656554e02b6649e8430c215fb P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:29.863229 2379 tablet_replica.cc:331] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:29.863569 2379 raft_consensus.cc:2241] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:29.823817 848 leader_election.cc:343] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.864809 1170 tablet_bootstrap.cc:492] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:29.865031 848 leader_election.cc:343] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.866798 1170 tablet_bootstrap.cc:654] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:29.866777 848 leader_election.cc:343] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.866953 848 leader_election.cc:343] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.867130 848 leader_election.cc:343] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.868081 2428 tablet.cc:1722] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:29.789955 1169 ts_tablet_manager.cc:1397] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.001s
I20250905 19:52:29.795485 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f52ff57fb1a4ea8a1ad4d5f0480c7f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.868853 1045 raft_consensus.cc:2466] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.796103 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.869278 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.869375 1044 raft_consensus.cc:2466] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.796962 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.869684 1051 raft_consensus.cc:2466] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.869748 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.797907 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.869976 1041 raft_consensus.cc:3058] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.826890 1171 tablet_bootstrap.cc:492] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.828594 2413 raft_consensus.cc:491] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.870088 2072 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38687
I20250905 19:52:29.870136 2413 raft_consensus.cc:513] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.870666 2413 leader_election.cc:290] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.870863 1171 ts_tablet_manager.cc:1397] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.271s user 0.004s sys 0.000s
I20250905 19:52:29.871496 1171 raft_consensus.cc:357] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.871632 1171 raft_consensus.cc:383] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.871676 1171 raft_consensus.cc:738] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.871913 1171 consensus_queue.cc:260] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.872496 2072 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-0/data/info.pb
I20250905 19:52:29.872509 1171 ts_tablet_manager.cc:1428] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:29.872689 1041 raft_consensus.cc:2466] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.872794 1171 tablet_bootstrap.cc:492] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.873042 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu as pid 2072
I20250905 19:52:29.798167 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.873291 1040 raft_consensus.cc:3058] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.873417 2491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38687 every 8 connection(s)
I20250905 19:52:29.874197 1171 tablet_bootstrap.cc:654] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.875593 1849 consensus_queue.cc:1035] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:29.798923 1038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc95a83ca872495894ef439cdda4e364" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.876174 1038 raft_consensus.cc:3058] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.876848 2379 raft_consensus.cc:2270] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.877588 2379 ts_tablet_manager.cc:1905] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:29.878226 712 leader_election.cc:343] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:29.878333 712 leader_election.cc:304] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:29.879482 1038 raft_consensus.cc:2466] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.799157 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:29.800081 2407 tablet.cc:1722] T 6e89d23c4cb64f568b25f92a278b9d5e P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.879813 1037 raft_consensus.cc:3058] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.811231 1798 raft_consensus.cc:491] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.881299 1798 raft_consensus.cc:513] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.881939 1798 leader_election.cc:290] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.828147 986 leader_election.cc:304] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:29.882273 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 54181.1us
I20250905 19:52:29.882488 986 leader_election.cc:304] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.882755 2386 raft_consensus.cc:2802] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.882829 2386 raft_consensus.cc:491] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.882884 2386 raft_consensus.cc:3058] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:29.882874 986 leader_election.cc:343] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.882992 986 leader_election.cc:304] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.883325 1849 raft_consensus.cc:2747] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:29.883366 986 leader_election.cc:343] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.883451 986 leader_election.cc:304] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.883744 2163 raft_consensus.cc:2802] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.883818 2163 raft_consensus.cc:491] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.883890 2163 raft_consensus.cc:3058] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.884586 2407 tablet.cc:1722] T 61667a379974487195f81043683b9a24 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:29.884747 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc95a83ca872495894ef439cdda4e364" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.885576 2386 raft_consensus.cc:513] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.886197 2386 leader_election.cc:290] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.886379 2386 raft_consensus.cc:2747] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.886528 2386 raft_consensus.cc:2747] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.886564 2163 raft_consensus.cc:513] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.886765 2376 ts_tablet_manager.cc:1918] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:29.886870 2376 log.cc:1199] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:29.886859 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a5af8232d31240ff855b79dcea29396b (table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]) successfully deleted
I20250905 19:52:29.887102 2163 leader_election.cc:290] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:29.887238 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 06850faa25de4b3d83f6bf98ccd370f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 06850faa25de4b3d83f6bf98ccd370f0 already in progress: creating tablet
I20250905 19:52:29.887887 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.890264 1037 raft_consensus.cc:2466] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:29.891957 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239539ac19124de192cadae494720a2b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.895298 1040 raft_consensus.cc:2466] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:29.896018 984 leader_election.cc:343] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.853539 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.896510 2432 tablet.cc:1722] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:29.899619 2379 ts_tablet_manager.cc:1918] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:29.899777 2379 log.cc:1199] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/36f966bf512d4cbe9ae406386db31533
I20250905 19:52:29.900238 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:29.900735 984 leader_election.cc:343] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.902957 2407 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.903327 1169 raft_consensus.cc:357] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.903533 1169 raft_consensus.cc:383] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.903586 1169 raft_consensus.cc:738] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.907881 1849 raft_consensus.cc:491] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.908089 1849 raft_consensus.cc:513] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:29.908380 986 leader_election.cc:343] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.908488 986 leader_election.cc:304] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:29.908867 986 leader_election.cc:343] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.908993 986 leader_election.cc:304] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.909233 1798 raft_consensus.cc:2747] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.909369 1798 raft_consensus.cc:2747] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.908205 712 leader_election.cc:304] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:29.910761 2160 raft_consensus.cc:2802] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.910899 2160 raft_consensus.cc:695] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:29.911026 1798 raft_consensus.cc:491] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.911134 1798 raft_consensus.cc:513] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.911165 2160 consensus_queue.cc:237] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.911665 1798 leader_election.cc:290] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.912292 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3a50526bb604a218649b24cbf65a65e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.912336 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.913070 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:29.913280 913 raft_consensus.cc:3058] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:29.913343 984 leader_election.cc:343] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.915122 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd9380e8ffc432da4a684934c2df68f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.915299 1041 raft_consensus.cc:2466] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:29.916162 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.916643 913 raft_consensus.cc:2466] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:29.917232 984 leader_election.cc:343] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.917683 1849 leader_election.cc:290] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.917919 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ba161282480e4736889559a2e37b195f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.919421 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:29.924146 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239539ac19124de192cadae494720a2b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:29.924880 913 raft_consensus.cc:3058] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.926172 1169 consensus_queue.cc:260] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.927040 1169 ts_tablet_manager.cc:1428] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.025s user 0.001s sys 0.001s
I20250905 19:52:29.927412 1169 tablet_bootstrap.cc:492] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:29.931305 913 raft_consensus.cc:2466] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:29.933275 986 leader_election.cc:343] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.933534 986 leader_election.cc:304] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.933866 1798 raft_consensus.cc:2747] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.934126 986 leader_election.cc:304] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.934459 1798 raft_consensus.cc:2802] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.934573 1798 raft_consensus.cc:695] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:29.934852 1798 consensus_queue.cc:237] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.939436 2163 raft_consensus.cc:491] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:29.939581 2163 raft_consensus.cc:513] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.940197 2163 leader_election.cc:290] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:29.941407 774 raft_consensus.cc:3058] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.942265 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:29.942828 984 leader_election.cc:343] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.946950 776 raft_consensus.cc:3058] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.951247 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd9380e8ffc432da4a684934c2df68f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:29.931847 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3a50526bb604a218649b24cbf65a65e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:29.953919 849 leader_election.cc:304] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.952509 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:29.954355 986 leader_election.cc:343] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.954555 986 leader_election.cc:304] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.955147 2496 raft_consensus.cc:2747] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.955353 986 leader_election.cc:304] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:29.957109 848 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.957554 848 leader_election.cc:343] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.957736 848 consensus_peers.cc:489] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:29.957913 848 leader_election.cc:343] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.958074 848 leader_election.cc:343] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:29.958252 848 consensus_peers.cc:489] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:29.958397 848 consensus_peers.cc:489] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:29.959951 2422 raft_consensus.cc:2802] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.960098 2422 raft_consensus.cc:491] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.960163 2422 raft_consensus.cc:3058] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.961915 2376 ts_tablet_manager.cc:1939] T 98e5ca7bd3e84916a11e01afbf48171d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:29.962553 2376 tablet_replica.cc:331] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:29.962503 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 98e5ca7bd3e84916a11e01afbf48171d (table default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff [id=8f425647eea7481181ba823eb955d1e1]) successfully deleted
I20250905 19:52:29.962730 2376 raft_consensus.cc:2241] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:29.962985 2376 raft_consensus.cc:2270] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.963574 2376 ts_tablet_manager.cc:1905] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.963713 2422 raft_consensus.cc:513] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.964578 2422 leader_election.cc:290] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.965126 849 leader_election.cc:304] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.965541 849 leader_election.cc:304] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.965811 2422 raft_consensus.cc:2762] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20250905 19:52:29.966022 2422 raft_consensus.cc:2802] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:29.966125 2422 raft_consensus.cc:491] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:29.966151 2379 ts_tablet_manager.cc:1939] T 36f966bf512d4cbe9ae406386db31533 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:29.966243 2422 raft_consensus.cc:3058] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:29.968871 1171 tablet_bootstrap.cc:492] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.968890 2422 raft_consensus.cc:513] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.969744 2422 leader_election.cc:290] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.969980 1171 ts_tablet_manager.cc:1397] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.000s
I20250905 19:52:29.970849 1171 raft_consensus.cc:357] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.971042 1171 raft_consensus.cc:383] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.971115 1171 raft_consensus.cc:738] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.971319 1171 consensus_queue.cc:260] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.971791 1171 ts_tablet_manager.cc:1428] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:29.972219 1171 tablet_bootstrap.cc:492] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:29.972739 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:29.973037 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:29.973605 1171 tablet_bootstrap.cc:654] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.978945 2379 tablet_replica.cc:331] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:29.979280 2379 raft_consensus.cc:2241] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:29.979509 2379 raft_consensus.cc:2270] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:29.978957 1170 tablet_bootstrap.cc:492] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:29.979772 1170 ts_tablet_manager.cc:1397] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.116s user 0.007s sys 0.000s
I20250905 19:52:29.980708 1170 raft_consensus.cc:357] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.981372 1170 raft_consensus.cc:383] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.981490 1170 raft_consensus.cc:738] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:29.977351 848 leader_election.cc:343] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.982127 1170 consensus_queue.cc:260] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.982820 1170 ts_tablet_manager.cc:1428] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:29.983177 1170 tablet_bootstrap.cc:492] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:29.984215 2496 raft_consensus.cc:2802] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.984354 2496 raft_consensus.cc:695] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
W20250905 19:52:29.984383 986 leader_election.cc:343] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:29.984490 986 leader_election.cc:304] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.984611 2496 consensus_queue.cc:237] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.985082 1170 tablet_bootstrap.cc:654] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:29.985935 2163 raft_consensus.cc:2747] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:29.986476 849 leader_election.cc:304] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:29.986750 2422 raft_consensus.cc:2802] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:29.986833 2379 ts_tablet_manager.cc:1905] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:29.986940 2422 raft_consensus.cc:695] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:29.987210 2422 consensus_queue.cc:237] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.987632 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:29.990959 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f52ff57fb1a4ea8a1ad4d5f0480c7f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.992058 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:29.993232 2108 raft_consensus.cc:513] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.994002 2108 leader_election.cc:290] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:29.994709 2432 tablet.cc:1722] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:29.996570 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:29.998726 1171 tablet_bootstrap.cc:492] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:29.998853 1171 ts_tablet_manager.cc:1397] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.000s
I20250905 19:52:29.999553 1171 raft_consensus.cc:357] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:29.999712 1171 raft_consensus.cc:383] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:29.999763 1171 raft_consensus.cc:738] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:29.999940 1171 consensus_queue.cc:260] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.000432 1171 ts_tablet_manager.cc:1428] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:30.000692 1171 tablet_bootstrap.cc:492] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.002079 1171 tablet_bootstrap.cc:654] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.006083 2492 sys_catalog.cc:263] Verifying existing consensus state
I20250905 19:52:30.011134 849 leader_election.cc:304] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.011510 1170 tablet_bootstrap.cc:492] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.011540 849 leader_election.cc:304] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.011844 2443 raft_consensus.cc:2802] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.011873 849 leader_election.cc:304] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.011986 2443 raft_consensus.cc:695] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.012271 2443 consensus_queue.cc:237] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.012715 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.019196 2492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Bootstrap starting.
I20250905 19:52:30.038672 1169 tablet_bootstrap.cc:700] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent opening tablet: real 0.110s user 0.001s sys 0.000s
I20250905 19:52:30.039310 1169 tablet_bootstrap.cc:654] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.043908 2444 raft_consensus.cc:2802] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.044131 2444 raft_consensus.cc:695] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.011663 1170 ts_tablet_manager.cc:1397] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250905 19:52:30.044420 2444 consensus_queue.cc:237] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.045653 2109 raft_consensus.cc:2802] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.045780 2109 raft_consensus.cc:695] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.045867 1170 raft_consensus.cc:357] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.046041 1170 raft_consensus.cc:383] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.045979 2109 consensus_queue.cc:237] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.046090 1170 raft_consensus.cc:738] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.050148 2379 ts_tablet_manager.cc:1918] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.050292 2379 log.cc:1199] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:30.050709 2379 ts_tablet_manager.cc:1939] T baefaa1f042348c1874e306f1e5a8781 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.052562 1170 consensus_queue.cc:260] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.066526 1170 ts_tablet_manager.cc:1428] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.022s user 0.001s sys 0.001s
I20250905 19:52:30.066713 1798 raft_consensus.cc:491] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.066879 1170 tablet_bootstrap.cc:492] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.066987 1798 raft_consensus.cc:513] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.070261 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.070941 984 leader_election.cc:343] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.071137 1170 tablet_bootstrap.cc:654] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:30.071461 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 47834.7us
W20250905 19:52:30.071767 848 leader_election.cc:343] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.072269 848 leader_election.cc:343] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.073452 848 leader_election.cc:343] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.075855 2095 raft_consensus.cc:491] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.078409 1798 leader_election.cc:290] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:30.079020 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:30.079460 986 leader_election.cc:343] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.079579 986 leader_election.cc:304] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.080329 1798 raft_consensus.cc:2747] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.081089 1171 tablet_bootstrap.cc:492] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.081215 1171 ts_tablet_manager.cc:1397] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.081s user 0.015s sys 0.013s
I20250905 19:52:30.081933 1171 raft_consensus.cc:357] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.082104 1171 raft_consensus.cc:383] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.082149 1171 raft_consensus.cc:738] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.082312 1171 consensus_queue.cc:260] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:30.082386 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e63b27ff4e434d76a8280f90c22f4082 consensus state has no leader
I20250905 19:52:30.082536 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 36f966bf512d4cbe9ae406386db31533 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]) successfully deleted
I20250905 19:52:30.082773 1171 ts_tablet_manager.cc:1428] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
W20250905 19:52:30.082792 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 98e5ca7bd3e84916a11e01afbf48171d because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 98e5ca7bd3e84916a11e01afbf48171d already in progress: creating tablet
I20250905 19:52:30.082979 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet baefaa1f042348c1874e306f1e5a8781 (table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]) successfully deleted
I20250905 19:52:30.083053 1171 tablet_bootstrap.cc:492] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:30.083302 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 71445.5us
I20250905 19:52:30.083794 2108 consensus_queue.cc:1035] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.085728 1171 tablet_bootstrap.cc:654] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.089923 849 leader_election.cc:304] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.090175 2108 raft_consensus.cc:2802] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.090243 2108 raft_consensus.cc:491] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.090297 2108 raft_consensus.cc:3058] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.092279 2095 raft_consensus.cc:513] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.092936 2095 leader_election.cc:290] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.093016 2108 raft_consensus.cc:513] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.093484 2108 leader_election.cc:290] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.095115 1170 tablet_bootstrap.cc:492] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.095264 1170 ts_tablet_manager.cc:1397] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.028s user 0.002s sys 0.002s
I20250905 19:52:30.095352 2376 ts_tablet_manager.cc:1918] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.095295 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:30.095789 2376 log.cc:1199] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a5af8232d31240ff855b79dcea29396b
I20250905 19:52:30.096160 2376 ts_tablet_manager.cc:1939] T a5af8232d31240ff855b79dcea29396b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.096087 1170 raft_consensus.cc:357] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.096307 1170 raft_consensus.cc:383] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.096371 1170 raft_consensus.cc:738] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.096485 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:30.096992 2379 tablet_replica.cc:331] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.097157 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:30.097230 2379 raft_consensus.cc:2241] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.097385 2379 raft_consensus.cc:2270] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.097353 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a5af8232d31240ff855b79dcea29396b (table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]) successfully deleted
I20250905 19:52:30.097678 2376 tablet_replica.cc:331] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.097910 2379 ts_tablet_manager.cc:1905] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.097985 2376 raft_consensus.cc:2241] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:30.098044 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:30.098304 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:30.098336 2376 raft_consensus.cc:2270] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.098815 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:30.098918 2376 ts_tablet_manager.cc:1905] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.099334 1170 consensus_queue.cc:260] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.100442 1170 ts_tablet_manager.cc:1428] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:30.100711 1170 tablet_bootstrap.cc:492] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.102273 712 leader_election.cc:304] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:30.102792 2160 raft_consensus.cc:2747] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.105672 1171 tablet_bootstrap.cc:492] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.106221 1171 ts_tablet_manager.cc:1397] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.007s
I20250905 19:52:30.107000 1171 raft_consensus.cc:357] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.107185 1171 raft_consensus.cc:383] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.107250 1171 raft_consensus.cc:738] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.107429 1171 consensus_queue.cc:260] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.107913 1171 ts_tablet_manager.cc:1428] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:30.108318 1171 tablet_bootstrap.cc:492] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.109910 1171 tablet_bootstrap.cc:654] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.116261 1171 tablet_bootstrap.cc:492] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.116384 1171 ts_tablet_manager.cc:1397] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.002s
I20250905 19:52:30.117081 1171 raft_consensus.cc:357] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.117137 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf448f9abc348df8d2d2501420d8f45" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.117230 1171 raft_consensus.cc:383] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.117274 1171 raft_consensus.cc:738] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.117426 1171 consensus_queue.cc:260] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.117748 1171 ts_tablet_manager.cc:1428] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:30.118007 1171 tablet_bootstrap.cc:492] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.118098 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd9380e8ffc432da4a684934c2df68f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:30.118218 2376 ts_tablet_manager.cc:1918] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.118297 2376 log.cc:1199] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:30.118695 2376 ts_tablet_manager.cc:1939] T baefaa1f042348c1874e306f1e5a8781 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.118927 1170 tablet_bootstrap.cc:654] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.119783 1171 tablet_bootstrap.cc:654] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.122684 2376 tablet_replica.cc:331] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.122926 2376 raft_consensus.cc:2241] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:30.123262 2376 raft_consensus.cc:2270] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.125480 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3f52ff57fb1a4ea8a1ad4d5f0480c7f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.125682 1050 raft_consensus.cc:3058] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.127206 2376 ts_tablet_manager.cc:1905] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.128207 2109 raft_consensus.cc:491] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.128343 2109 raft_consensus.cc:513] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.128701 1050 raft_consensus.cc:2466] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.128923 2109 leader_election.cc:290] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.131062 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.131346 1050 raft_consensus.cc:3058] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.134259 1050 raft_consensus.cc:2466] T 4dd2a96b8180438fbc722be1c18a0189 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.134585 2379 ts_tablet_manager.cc:1918] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:30.134637 2376 ts_tablet_manager.cc:1918] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.134735 2379 log.cc:1199] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/0edac849e8834894be2d4ff323edd13d
I20250905 19:52:30.134734 2376 log.cc:1199] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8edf4b54974c427a855eee5792d75292
I20250905 19:52:30.135071 2376 ts_tablet_manager.cc:1939] T 8edf4b54974c427a855eee5792d75292 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.135114 2379 ts_tablet_manager.cc:1939] T 0edac849e8834894be2d4ff323edd13d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.137552 1171 tablet_bootstrap.cc:492] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.137686 1171 ts_tablet_manager.cc:1397] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.003s
I20250905 19:52:30.138439 1171 raft_consensus.cc:357] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.138603 1171 raft_consensus.cc:383] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.138654 1171 raft_consensus.cc:738] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.138881 1171 consensus_queue.cc:260] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.144433 1171 ts_tablet_manager.cc:1428] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:30.144814 1171 tablet_bootstrap.cc:492] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.148805 1171 tablet_bootstrap.cc:654] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.149868 1170 tablet_bootstrap.cc:492] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.150039 1170 ts_tablet_manager.cc:1397] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250905 19:52:30.151145 1170 raft_consensus.cc:357] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.151372 1170 raft_consensus.cc:383] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.151423 1170 raft_consensus.cc:738] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.151698 1170 consensus_queue.cc:260] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:30.123656 848 leader_election.cc:343] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.152223 848 leader_election.cc:343] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.152349 1170 ts_tablet_manager.cc:1428] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:30.152705 1170 tablet_bootstrap.cc:492] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.152947 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:30.154501 1170 tablet_bootstrap.cc:654] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:30.157083 848 consensus_peers.cc:489] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.157312 2376 tablet_replica.cc:331] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.157475 2376 raft_consensus.cc:2241] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.157605 2376 raft_consensus.cc:2270] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.160033 2376 ts_tablet_manager.cc:1905] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.161715 2379 tablet_replica.cc:331] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.161995 2379 raft_consensus.cc:2241] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:30.162315 2379 pending_rounds.cc:70] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
I20250905 19:52:30.168027 2379 pending_rounds.cc:77] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089578983649280 op_type: NO_OP noop_request { }
I20250905 19:52:30.168273 2379 raft_consensus.cc:2887] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:30.168440 2379 raft_consensus.cc:2270] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.164170 774 raft_consensus.cc:3058] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.169263 2379 ts_tablet_manager.cc:1905] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.176348 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2350bd08f977424ba94ee8fa01f66efa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.176899 848 leader_election.cc:343] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.177937 2464 raft_consensus.cc:491] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.178064 2464 raft_consensus.cc:513] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.179234 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.179800 2464 leader_election.cc:290] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.185942 2376 ts_tablet_manager.cc:1918] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.186094 2376 log.cc:1199] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:30.186544 2376 ts_tablet_manager.cc:1939] T b0702063ec1c4f6fba58e6d4f3912212 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.191104 2379 ts_tablet_manager.cc:1918] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.192608 1169 tablet_bootstrap.cc:492] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.192808 1169 ts_tablet_manager.cc:1397] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.265s user 0.004s sys 0.000s
I20250905 19:52:30.192854 2379 log.cc:1199] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:30.193507 2379 ts_tablet_manager.cc:1939] T 8ed6109bdf8f4caf8c223a42401b94ab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.193786 1169 raft_consensus.cc:357] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.193975 1169 raft_consensus.cc:383] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.194056 1169 raft_consensus.cc:738] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.194430 1169 consensus_queue.cc:260] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.195034 2376 tablet_replica.cc:331] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.195079 1169 ts_tablet_manager.cc:1428] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20250905 19:52:30.195389 2376 raft_consensus.cc:2241] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.196498 2376 raft_consensus.cc:2270] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.196519 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.196743 1039 raft_consensus.cc:2466] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.198447 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5bd9380e8ffc432da4a684934c2df68f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.198647 1041 raft_consensus.cc:3058] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.199220 2482 tablet.cc:1722] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 5
I20250905 19:52:30.199826 2376 ts_tablet_manager.cc:1905] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.201352 1041 raft_consensus.cc:2466] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.201963 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf448f9abc348df8d2d2501420d8f45" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.202174 1039 raft_consensus.cc:2466] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.203387 2464 raft_consensus.cc:491] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.203634 2464 raft_consensus.cc:513] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.205041 2464 leader_election.cc:290] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:30.205453 711 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:30.208302 774 raft_consensus.cc:3058] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.208588 2379 tablet_replica.cc:331] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.209156 2379 raft_consensus.cc:2241] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.209744 2379 raft_consensus.cc:2270] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:30.210057 848 consensus_peers.cc:489] T 3587780819704008b7f4ee6abc714db6 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.210479 2464 raft_consensus.cc:491] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.210695 2464 raft_consensus.cc:513] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.211679 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:30.211956 2464 leader_election.cc:290] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.212781 2379 ts_tablet_manager.cc:1905] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.221038 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "54628965b82e470d811eff95ffc56c5f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.225268 2376 ts_tablet_manager.cc:1918] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.225485 2376 log.cc:1199] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ab4f69c45295422f85b310aa55220e9b
I20250905 19:52:30.225975 2376 ts_tablet_manager.cc:1939] T ab4f69c45295422f85b310aa55220e9b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.227844 2376 tablet_replica.cc:331] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.228193 2376 raft_consensus.cc:2241] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.228447 2376 raft_consensus.cc:2270] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.121671 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet baefaa1f042348c1874e306f1e5a8781 (table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]) successfully deleted
W20250905 19:52:30.228828 1484 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 107151us
I20250905 19:52:30.229066 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8edf4b54974c427a855eee5792d75292 (table default.Table_93a9ab7d352a4a3da35219aa6f20f432 [id=b5acc86aee9545d99f8d8332c3c76a8e]) successfully deleted
I20250905 19:52:30.229238 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet b0702063ec1c4f6fba58e6d4f3912212 (table default.Table_46df29d8bee2462fb784be95829c6ad0 [id=3ed507a38d1049fa893efa7be65f3609]) successfully deleted
I20250905 19:52:30.229363 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ab4f69c45295422f85b310aa55220e9b (table default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]) successfully deleted
I20250905 19:52:30.230070 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
W20250905 19:52:30.099241 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 14d5eb8bec074d2c8766fe7b159031e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14d5eb8bec074d2c8766fe7b159031e5 already in progress: creating tablet
W20250905 19:52:30.231220 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 132022us
I20250905 19:52:30.232236 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 0edac849e8834894be2d4ff323edd13d (table default.Table_a39be9af778c4fa49c2376cb4bd7397b [id=e9ed839c21da4467a0f165e7fedbe966]) successfully deleted
I20250905 19:52:30.232486 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8ed6109bdf8f4caf8c223a42401b94ab (table default.Table_68f98d1c0de04ff9a6ad5188ab275ec9 [id=711f36b2be7c4a368fcb3106e691b133]) successfully deleted
I20250905 19:52:30.234275 774 raft_consensus.cc:3058] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.236516 2376 ts_tablet_manager.cc:1905] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.237967 2379 ts_tablet_manager.cc:1918] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:30.242100 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:30.242463 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:30.242779 1539 catalog_manager.cc:5582] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:30.243173 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:30.243520 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "091ef981d6b54756830927ea111e78aa" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.243767 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:30.245620 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.253770 2376 ts_tablet_manager.cc:1918] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.253958 2376 log.cc:1199] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:30.254405 2376 ts_tablet_manager.cc:1939] T 0267ac3e3f1a45978807737eca5d91ef P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.258311 2376 tablet_replica.cc:331] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.258358 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 0267ac3e3f1a45978807737eca5d91ef (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]) successfully deleted
I20250905 19:52:30.258646 2376 raft_consensus.cc:2241] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.259048 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ac5c88cf8d15433a940e937f0a225582" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.264005 1169 tablet_bootstrap.cc:492] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.265758 1169 tablet_bootstrap.cc:654] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.276782 2529 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:30.277015 2529 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:30.277915 2529 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:30.278625 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 636 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
I20250905 19:52:30.287020 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:30.287477 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:30.287792 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:30.288379 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:30.288915 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:30.289299 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:30.289695 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:30.290234 2376 raft_consensus.cc:2270] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.290354 849 leader_election.cc:304] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.290622 2444 raft_consensus.cc:2802] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.290799 2444 raft_consensus.cc:695] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.290846 849 leader_election.cc:304] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.291082 2422 raft_consensus.cc:2802] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.291143 2376 ts_tablet_manager.cc:1905] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.291234 2422 raft_consensus.cc:695] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.291505 2422 consensus_queue.cc:237] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.297277 2444 consensus_queue.cc:237] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.298702 712 leader_election.cc:304] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:30.299218 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 49441.9us
I20250905 19:52:30.301690 1171 tablet_bootstrap.cc:492] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.301923 1171 ts_tablet_manager.cc:1397] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.157s user 0.030s sys 0.036s
I20250905 19:52:30.302208 2376 ts_tablet_manager.cc:1918] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.302400 2376 log.cc:1199] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/00d8144bb4f14358af539b7aae326918
I20250905 19:52:30.302842 2376 ts_tablet_manager.cc:1939] T 00d8144bb4f14358af539b7aae326918 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.303113 1171 raft_consensus.cc:357] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.303385 1171 raft_consensus.cc:383] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.303275 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 54) took 1037 ms. Trace:
I20250905 19:52:30.303511 1171 raft_consensus.cc:738] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.303668 2376 rpcz_store.cc:276] 0905 19:52:29.265443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.265812 (+ 369us) service_pool.cc:225] Handling call
0905 19:52:30.258256 (+992444us) ts_tablet_manager.cc:1145] Deleting tablet 00d8144bb4f14358af539b7aae326918
0905 19:52:30.258267 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.295323 (+ 37056us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.302054 (+ 6731us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.303195 (+ 1141us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":31270,"tablet-delete.queue_time_us":992258}
I20250905 19:52:30.304477 2376 tablet_replica.cc:331] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.304785 2376 raft_consensus.cc:2241] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:30.305236 2376 raft_consensus.cc:2270] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.305905 2376 ts_tablet_manager.cc:1905] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.307829 849 leader_election.cc:304] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.308199 2108 raft_consensus.cc:2802] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.308215 2464 raft_consensus.cc:2747] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.308329 2108 raft_consensus.cc:491] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.308462 2108 raft_consensus.cc:3058] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.308895 2100 raft_consensus.cc:491] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.309774 2100 raft_consensus.cc:513] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.310514 2100 leader_election.cc:290] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.311059 2108 raft_consensus.cc:513] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.313442 2108 leader_election.cc:290] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:30.315237 711 leader_election.cc:343] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:30.318277 2376 ts_tablet_manager.cc:1918] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.318686 2376 log.cc:1199] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/578a6b9224cc4d8585d6c4b241762e1b
I20250905 19:52:30.319176 2376 ts_tablet_manager.cc:1939] T 578a6b9224cc4d8585d6c4b241762e1b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.319532 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 58) took 1053 ms. Trace:
I20250905 19:52:30.319693 2376 rpcz_store.cc:276] 0905 19:52:29.265974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.266041 (+ 67us) service_pool.cc:225] Handling call
0905 19:52:30.304389 (+1038348us) ts_tablet_manager.cc:1145] Deleting tablet 578a6b9224cc4d8585d6c4b241762e1b
0905 19:52:30.304399 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.315198 (+ 10799us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.318154 (+ 2956us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.319499 (+ 1345us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1038188}
I20250905 19:52:30.320503 2376 tablet_replica.cc:331] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.320814 2376 raft_consensus.cc:2241] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.321527 1169 tablet_bootstrap.cc:492] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.321627 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:30.321672 2413 consensus_queue.cc:1035] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.322851 2376 raft_consensus.cc:2270] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.323570 1169 ts_tablet_manager.cc:1397] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.000s
I20250905 19:52:30.324034 2376 ts_tablet_manager.cc:1905] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:30.322247 848 leader_election.cc:343] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.322165 849 leader_election.cc:304] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.324707 2413 raft_consensus.cc:2802] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.324867 2413 raft_consensus.cc:695] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.325138 2413 consensus_queue.cc:237] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.303828 1171 consensus_queue.cc:260] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.328198 1171 ts_tablet_manager.cc:1428] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.026s user 0.003s sys 0.000s
I20250905 19:52:30.328506 1171 tablet_bootstrap.cc:492] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.330379 1171 tablet_bootstrap.cc:654] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.304445 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 00d8144bb4f14358af539b7aae326918 (table default.Table_9ae350716fe144629f1f50ebee119d72 [id=92a9305bd65247c7912a68913af4e967]) successfully deleted
I20250905 19:52:30.331804 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 578a6b9224cc4d8585d6c4b241762e1b (table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]) successfully deleted
I20250905 19:52:30.332515 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:30.332784 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:30.333001 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:30.333644 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:30.334064 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:30.334504 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:30.334880 2379 log.cc:1199] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:30.335177 1539 catalog_manager.cc:5582] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:30.335889 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:30.336354 849 leader_election.cc:304] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.336570 2379 ts_tablet_manager.cc:1939] T 757a7d3fba8d43dda36474539b7296ba P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.338994 2108 raft_consensus.cc:2802] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.339136 2108 raft_consensus.cc:491] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.339246 2108 raft_consensus.cc:3058] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.341949 2108 raft_consensus.cc:513] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.343025 2108 leader_election.cc:290] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.343647 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf448f9abc348df8d2d2501420d8f45" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:30.349138 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:30.339172 1170 tablet_bootstrap.cc:492] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
W20250905 19:52:30.352274 848 leader_election.cc:343] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.357028 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:30.358189 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:30.359418 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:30.361894 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:30.362582 2095 raft_consensus.cc:491] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.368516 1169 raft_consensus.cc:357] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.368742 1169 raft_consensus.cc:383] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.370431 1169 raft_consensus.cc:738] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.370656 1169 consensus_queue.cc:260] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.373790 1169 ts_tablet_manager.cc:1428] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.050s user 0.002s sys 0.000s
I20250905 19:52:30.375255 1169 tablet_bootstrap.cc:492] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.376511 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 61) took 1013 ms. Trace:
I20250905 19:52:30.376724 1170 ts_tablet_manager.cc:1397] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.204s user 0.009s sys 0.010s
I20250905 19:52:30.377620 1170 raft_consensus.cc:357] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.377925 1170 raft_consensus.cc:383] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.378044 1170 raft_consensus.cc:738] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.379376 1170 consensus_queue.cc:260] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.381258 1169 tablet_bootstrap.cc:654] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.381946 2379 rpcz_store.cc:276] 0905 19:52:29.337323 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337372 (+ 49us) service_pool.cc:225] Handling call
0905 19:52:30.208179 (+870807us) ts_tablet_manager.cc:1145] Deleting tablet 757a7d3fba8d43dda36474539b7296ba
0905 19:52:30.208194 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.217016 (+ 8822us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.237201 (+ 20185us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.351274 (+114073us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":865815}
I20250905 19:52:30.382493 2379 tablet_replica.cc:331] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.382773 2379 raft_consensus.cc:2241] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:30.383584 2379 raft_consensus.cc:2270] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.384227 2379 ts_tablet_manager.cc:1905] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.380079 1170 ts_tablet_manager.cc:1428] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:30.392030 1170 tablet_bootstrap.cc:492] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:30.394984 712 consensus_peers.cc:489] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.395458 2160 consensus_queue.cc:1035] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.396696 712 leader_election.cc:304] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:30.397133 2160 raft_consensus.cc:2747] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:30.397608 712 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:30.397805 712 leader_election.cc:304] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:30.398275 712 leader_election.cc:304] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:30.398551 2160 raft_consensus.cc:2747] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.398875 2160 raft_consensus.cc:2747] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.399551 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 757a7d3fba8d43dda36474539b7296ba (table default.Table_a5a47a603b674a41b47e7bf4b55b6a1b [id=106f635aee6b4cd4a7f7b1af9088a286]) successfully deleted
I20250905 19:52:30.400218 2100 raft_consensus.cc:491] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.400357 2100 raft_consensus.cc:513] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.400977 2100 leader_election.cc:290] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:30.402293 711 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
W20250905 19:52:30.408226 849 leader_election.cc:343] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:30.408391 849 leader_election.cc:304] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.408730 1169 tablet_bootstrap.cc:492] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.408897 1169 ts_tablet_manager.cc:1397] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.034s user 0.002s sys 0.005s
I20250905 19:52:30.409932 1169 raft_consensus.cc:357] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.410135 1169 raft_consensus.cc:383] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.410228 1169 raft_consensus.cc:738] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.411474 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.411666 1036 raft_consensus.cc:3058] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.411697 2109 consensus_queue.cc:1035] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20250905 19:52:30.412201 2444 raft_consensus.cc:2747] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.412348 2095 raft_consensus.cc:513] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.412879 2095 leader_election.cc:290] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.413410 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31e332cdfedd4ff6b5026baa73e2caf3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.415521 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dcf448f9abc348df8d2d2501420d8f45" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
W20250905 19:52:30.415555 848 leader_election.cc:343] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.415702 1037 raft_consensus.cc:3058] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.417263 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f3473785fff45d692bd50b1015ffe39" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.418120 909 raft_consensus.cc:2466] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:30.418386 1037 raft_consensus.cc:2466] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.419291 1036 raft_consensus.cc:2466] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.419764 2492 log.cc:826] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Log is configured to *not* fsync() on all Append() calls
I20250905 19:52:30.420401 1169 consensus_queue.cc:260] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.420990 1169 ts_tablet_manager.cc:1428] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250905 19:52:30.421321 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31e332cdfedd4ff6b5026baa73e2caf3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.421506 1040 raft_consensus.cc:2466] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.423017 2376 ts_tablet_manager.cc:1918] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.418179 1170 tablet_bootstrap.cc:654] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.428964 774 raft_consensus.cc:3058] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:30.429702 712 consensus_peers.cc:489] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.429987 712 leader_election.cc:304] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:30.430289 2100 raft_consensus.cc:2802] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.430387 2100 raft_consensus.cc:491] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.430459 2100 raft_consensus.cc:3058] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.433557 2100 raft_consensus.cc:513] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.434211 2100 leader_election.cc:290] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:30.434698 712 leader_election.cc:343] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:30.434919 712 leader_election.cc:304] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:30.435200 711 leader_election.cc:343] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:30.436348 2376 log.cc:1199] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:30.436828 2376 ts_tablet_manager.cc:1939] T 4d2ed1bc09c94526bc9662f2d4d21f8f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.437217 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 55) took 1171 ms. Trace:
I20250905 19:52:30.437358 2376 rpcz_store.cc:276] 0905 19:52:29.265538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.266224 (+ 686us) service_pool.cc:225] Handling call
0905 19:52:30.320426 (+1054202us) ts_tablet_manager.cc:1145] Deleting tablet 4d2ed1bc09c94526bc9662f2d4d21f8f
0905 19:52:30.320436 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.399874 (+ 79438us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.422895 (+ 23021us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.437169 (+ 14274us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1054035}
I20250905 19:52:30.437955 2376 tablet_replica.cc:331] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.437909 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 4d2ed1bc09c94526bc9662f2d4d21f8f (table default.Table_315b3dd493ac42fea5c100b3e58d9533 [id=70f8481033284889b47c81173fb2ef5c]) successfully deleted
I20250905 19:52:30.438243 2376 raft_consensus.cc:2241] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:30.430105 849 leader_election.cc:304] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.439052 2095 raft_consensus.cc:2802] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.439260 2095 raft_consensus.cc:695] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.439568 2095 consensus_queue.cc:237] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.441629 849 leader_election.cc:304] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.441900 2444 raft_consensus.cc:2802] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.442024 2444 raft_consensus.cc:695] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.442250 2444 consensus_queue.cc:237] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.446040 1171 tablet_bootstrap.cc:492] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.446396 1171 ts_tablet_manager.cc:1397] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.118s user 0.003s sys 0.006s
I20250905 19:52:30.447330 1171 raft_consensus.cc:357] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.447588 1171 raft_consensus.cc:383] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.447700 1171 raft_consensus.cc:738] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.448022 1171 consensus_queue.cc:260] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.448606 1171 ts_tablet_manager.cc:1428] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:30.448999 1171 tablet_bootstrap.cc:492] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.450451 849 leader_election.cc:304] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.451143 2109 raft_consensus.cc:2802] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.455747 1169 tablet_bootstrap.cc:492] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.458087 1169 tablet_bootstrap.cc:654] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.463004 2100 raft_consensus.cc:2747] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.469311 2109 raft_consensus.cc:491] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.469532 2109 raft_consensus.cc:3058] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.469837 2100 raft_consensus.cc:491] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.470008 2100 raft_consensus.cc:513] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.470804 2100 leader_election.cc:290] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.472608 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6f3473785fff45d692bd50b1015ffe39" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:30.476085 912 raft_consensus.cc:3058] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.479300 912 raft_consensus.cc:2466] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:30.480683 2379 ts_tablet_manager.cc:1918] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.481122 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.481506 2379 log.cc:1199] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/0267ac3e3f1a45978807737eca5d91ef
I20250905 19:52:30.483919 1170 tablet_bootstrap.cc:492] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.484278 1170 ts_tablet_manager.cc:1397] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.092s user 0.000s sys 0.009s
I20250905 19:52:30.485260 1169 tablet_bootstrap.cc:492] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.485466 1169 ts_tablet_manager.cc:1397] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.030s user 0.006s sys 0.006s
I20250905 19:52:30.486187 1170 raft_consensus.cc:357] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.486589 1170 raft_consensus.cc:383] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.486788 1170 raft_consensus.cc:738] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.486761 1169 raft_consensus.cc:357] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.486981 1169 raft_consensus.cc:383] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.487046 1169 raft_consensus.cc:738] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.487300 1169 consensus_queue.cc:260] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.487277 1170 consensus_queue.cc:260] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.487803 1169 ts_tablet_manager.cc:1428] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:30.488144 1169 tablet_bootstrap.cc:492] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.488144 712 leader_election.cc:304] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:30.489523 2100 raft_consensus.cc:2802] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.489727 2100 raft_consensus.cc:695] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:30.490038 2100 consensus_queue.cc:237] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.490306 1170 ts_tablet_manager.cc:1428] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:30.490604 2379 ts_tablet_manager.cc:1939] T 0267ac3e3f1a45978807737eca5d91ef P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.492350 1170 tablet_bootstrap.cc:492] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.493091 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 62) took 1155 ms. Trace:
I20250905 19:52:30.493695 2379 rpcz_store.cc:276] 0905 19:52:29.337445 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337513 (+ 68us) service_pool.cc:225] Handling call
0905 19:52:30.382424 (+1044911us) ts_tablet_manager.cc:1145] Deleting tablet 0267ac3e3f1a45978807737eca5d91ef
0905 19:52:30.382436 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.441843 (+ 59407us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.480078 (+ 38235us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.492696 (+ 12618us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1039025}
I20250905 19:52:30.496165 2379 tablet_replica.cc:331] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.497260 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 0267ac3e3f1a45978807737eca5d91ef (table default.Table_2dbd3207d30d4b20b44f6c0f2b03a13e [id=a53613dd2a0a430f83f027b67ee14b2e]) successfully deleted
I20250905 19:52:30.497961 2422 consensus_queue.cc:1035] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.499425 1169 tablet_bootstrap.cc:654] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.505529 1169 tablet_bootstrap.cc:492] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.505702 1169 ts_tablet_manager.cc:1397] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.002s
I20250905 19:52:30.498150 2379 raft_consensus.cc:2241] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.513293 1169 raft_consensus.cc:357] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.513562 1169 raft_consensus.cc:383] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.515364 1169 raft_consensus.cc:738] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.515625 1169 consensus_queue.cc:260] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.517094 1169 ts_tablet_manager.cc:1428] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.002s sys 0.001s
I20250905 19:52:30.517470 1169 tablet_bootstrap.cc:492] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.519609 1169 tablet_bootstrap.cc:654] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.532037 2496 raft_consensus.cc:491] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.532235 2496 raft_consensus.cc:513] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.532909 2496 leader_election.cc:290] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:30.533252 2379 raft_consensus.cc:2270] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.533569 1171 tablet_bootstrap.cc:654] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.533929 2379 ts_tablet_manager.cc:1905] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.537755 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b44c00d3b54af6b0752575e6898839" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.537765 2413 raft_consensus.cc:491] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.539456 2444 raft_consensus.cc:491] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:30.539870 986 leader_election.cc:343] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.540488 1170 tablet_bootstrap.cc:654] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.540623 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b44c00d3b54af6b0752575e6898839" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.541381 2376 raft_consensus.cc:2270] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:30.541532 984 leader_election.cc:343] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.541687 984 leader_election.cc:304] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.541846 1169 tablet_bootstrap.cc:492] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.541960 2496 raft_consensus.cc:2747] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.542019 1169 ts_tablet_manager.cc:1397] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250905 19:52:30.542860 1169 raft_consensus.cc:357] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.543043 1169 raft_consensus.cc:383] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.543123 1169 raft_consensus.cc:738] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.543366 1169 consensus_queue.cc:260] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.547528 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "de19c897d3314fecb17cedb3370eed93" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.547832 1169 ts_tablet_manager.cc:1428] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:30.548164 1169 tablet_bootstrap.cc:492] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.550189 1169 tablet_bootstrap.cc:654] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.571236 2496 raft_consensus.cc:491] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.571420 2496 raft_consensus.cc:513] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.571033 2379 ts_tablet_manager.cc:1918] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:30.572055 2496 leader_election.cc:290] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:30.580506 1025 tablet_service.cc:1468] Processing CreateTablet for tablet e77bdf9df6eb4470a3d22e96da952718 (DEFAULT_TABLE table=default.Table_96c633b175ac41309951506efe002443 [id=ae92837e2ac94949aac8822f084c0a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.581070 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77bdf9df6eb4470a3d22e96da952718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.573308 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.576040 749 tablet_service.cc:1468] Processing CreateTablet for tablet e77bdf9df6eb4470a3d22e96da952718 (DEFAULT_TABLE table=default.Table_96c633b175ac41309951506efe002443 [id=ae92837e2ac94949aac8822f084c0a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.587898 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77bdf9df6eb4470a3d22e96da952718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.590199 737 tablet_service.cc:1468] Processing CreateTablet for tablet 10fef848b9b746c5a0de6a8c3ca74f7a (DEFAULT_TABLE table=default.Table_94f58de97c5946149d23d635eda4d555 [id=656d24e0a3ad4ac78696df5aec2f279e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.590571 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fef848b9b746c5a0de6a8c3ca74f7a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:30.592283 984 leader_election.cc:343] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.592746 1169 tablet_bootstrap.cc:492] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.592862 1169 ts_tablet_manager.cc:1397] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.000s
I20250905 19:52:30.593611 1169 raft_consensus.cc:357] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.593775 1169 raft_consensus.cc:383] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.593825 1169 raft_consensus.cc:738] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.594046 1169 consensus_queue.cc:260] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.594632 1169 ts_tablet_manager.cc:1428] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:30.572894 2379 log.cc:1199] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/0d4204f61929498188c18529148a8d13
I20250905 19:52:30.594900 1169 tablet_bootstrap.cc:492] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.596455 1169 tablet_bootstrap.cc:654] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.598778 2379 ts_tablet_manager.cc:1939] T 0d4204f61929498188c18529148a8d13 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:30.598181 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 97109.5us
I20250905 19:52:30.603454 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 58) took 1266 ms. Trace:
I20250905 19:52:30.604878 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.605311 2100 raft_consensus.cc:491] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.605448 2100 raft_consensus.cc:513] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:30.605520 986 leader_election.cc:343] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.605660 986 leader_election.cc:304] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.606060 2496 raft_consensus.cc:2747] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.606678 2100 leader_election.cc:290] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:30.607616 711 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:30.605127 2379 rpcz_store.cc:276] 0905 19:52:29.336995 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337568 (+ 573us) service_pool.cc:225] Handling call
0905 19:52:30.495598 (+1158030us) ts_tablet_manager.cc:1145] Deleting tablet 0d4204f61929498188c18529148a8d13
0905 19:52:30.495610 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.559438 (+ 63828us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.569883 (+ 10445us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.603396 (+ 33513us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1151881}
I20250905 19:52:30.608425 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 0d4204f61929498188c18529148a8d13 (table default.Table_97c63bdbe93a48708207c1e9a96cb1bd [id=fc5c41a92d3e4bfc9c7d07fdd1ec598e]) successfully deleted
I20250905 19:52:30.609758 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 10fef848b9b746c5a0de6a8c3ca74f7a (DEFAULT_TABLE table=default.Table_94f58de97c5946149d23d635eda4d555 [id=656d24e0a3ad4ac78696df5aec2f279e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.610260 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fef848b9b746c5a0de6a8c3ca74f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.610226 2379 tablet_replica.cc:331] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.614619 2109 raft_consensus.cc:513] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.614751 2376 ts_tablet_manager.cc:1905] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.615116 2444 raft_consensus.cc:513] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.615269 2109 leader_election.cc:290] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.615605 2444 leader_election.cc:290] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.615016 2379 raft_consensus.cc:2241] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.617717 1170 tablet_bootstrap.cc:492] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.617846 1170 ts_tablet_manager.cc:1397] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.126s user 0.011s sys 0.000s
I20250905 19:52:30.618072 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (DEFAULT_TABLE table=default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.618433 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.618583 1170 raft_consensus.cc:357] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.618731 1170 raft_consensus.cc:383] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.618779 1170 raft_consensus.cc:738] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.619158 1170 consensus_queue.cc:260] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.620091 1170 ts_tablet_manager.cc:1428] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:30.620618 1170 tablet_bootstrap.cc:492] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.623231 1170 tablet_bootstrap.cc:654] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.624537 737 tablet_service.cc:1468] Processing CreateTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (DEFAULT_TABLE table=default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.625038 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.626154 2413 raft_consensus.cc:513] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.628152 2413 leader_election.cc:290] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.632941 2379 raft_consensus.cc:2270] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.634132 2379 ts_tablet_manager.cc:1905] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.638438 749 tablet_service.cc:1468] Processing CreateTablet for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 (DEFAULT_TABLE table=default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.638908 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.578982 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:30.647799 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 126545us
I20250905 19:52:30.654000 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 (DEFAULT_TABLE table=default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.654172 712 leader_election.cc:304] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:30.654528 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.654632 2100 raft_consensus.cc:2747] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.655618 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31e332cdfedd4ff6b5026baa73e2caf3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.655797 1036 raft_consensus.cc:3058] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.658516 1036 raft_consensus.cc:2466] T 31e332cdfedd4ff6b5026baa73e2caf3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.659271 1170 tablet_bootstrap.cc:492] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.659929 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.660225 1036 raft_consensus.cc:2466] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.661330 1170 ts_tablet_manager.cc:1397] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.000s
I20250905 19:52:30.662911 737 tablet_service.cc:1468] Processing CreateTablet for tablet 6b8bbbcac06944469e13f14ab1d80402 (DEFAULT_TABLE table=default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.663342 737 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8bbbcac06944469e13f14ab1d80402. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.672057 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 6b8bbbcac06944469e13f14ab1d80402 (DEFAULT_TABLE table=default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.672511 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8bbbcac06944469e13f14ab1d80402. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.673702 749 tablet_service.cc:1468] Processing CreateTablet for tablet 298a325f5e4645beac39eb59ddc0566d (DEFAULT_TABLE table=default.Table_44734742f0c948af9cce54466ff759c6 [id=11511883d6b046c7b155ff9687eb2c76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.674104 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298a325f5e4645beac39eb59ddc0566d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.676674 2108 raft_consensus.cc:491] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.676918 2108 raft_consensus.cc:513] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.677670 2108 leader_election.cc:290] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.680375 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 298a325f5e4645beac39eb59ddc0566d (DEFAULT_TABLE table=default.Table_44734742f0c948af9cce54466ff759c6 [id=11511883d6b046c7b155ff9687eb2c76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.680763 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298a325f5e4645beac39eb59ddc0566d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.678653 1170 raft_consensus.cc:357] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.683311 740 tablet_service.cc:1468] Processing CreateTablet for tablet 6edbc312fc4947a79e937dbc64d30b37 (DEFAULT_TABLE table=default.Table_56ae03b8d3eb4a58970a492506fcc6f1 [id=acf1b6990a594051ba94b43420cdeebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.683678 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edbc312fc4947a79e937dbc64d30b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.682906 1170 raft_consensus.cc:383] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.683894 1170 raft_consensus.cc:738] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.688926 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3315d3e9633e42ea8d7a6b5934cb0438" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.689150 1037 raft_consensus.cc:2466] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:30.689710 712 leader_election.cc:343] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:30.689850 712 leader_election.cc:304] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:30.687759 1170 consensus_queue.cc:260] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.690511 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 6edbc312fc4947a79e937dbc64d30b37 (DEFAULT_TABLE table=default.Table_56ae03b8d3eb4a58970a492506fcc6f1 [id=acf1b6990a594051ba94b43420cdeebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.690979 2100 raft_consensus.cc:2747] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.691617 739 tablet_service.cc:1468] Processing CreateTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 (DEFAULT_TABLE table=default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.692023 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f5d19f0216a4f069237e48c62ee7875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.693152 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 (DEFAULT_TABLE table=default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.694159 741 tablet_service.cc:1468] Processing CreateTablet for tablet bb2b5ce70b784c13aa64053b2d1fe541 (DEFAULT_TABLE table=default.Table_57684ee2f11b44129e0b3d35029151fc [id=d636c310314a405e8d4f99e56d94f44c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.694751 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2b5ce70b784c13aa64053b2d1fe541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.696035 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:30.697378 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edbc312fc4947a79e937dbc64d30b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.698493 2574 raft_consensus.cc:491] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.698619 2574 raft_consensus.cc:513] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.699136 2574 leader_election.cc:290] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.700738 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f5d19f0216a4f069237e48c62ee7875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.698752 849 leader_election.cc:304] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:30.702749 849 leader_election.cc:304] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:30.703132 849 leader_election.cc:343] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:30.703718 2095 raft_consensus.cc:2802] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.703771 849 leader_election.cc:304] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:30.703938 2443 raft_consensus.cc:2802] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.704010 2443 raft_consensus.cc:491] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.704070 2443 raft_consensus.cc:3058] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.703544 1170 ts_tablet_manager.cc:1428] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.041s user 0.003s sys 0.000s
I20250905 19:52:30.704965 2444 raft_consensus.cc:2802] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.705042 2444 raft_consensus.cc:491] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.705097 2444 raft_consensus.cc:3058] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.703840 2095 raft_consensus.cc:695] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.706137 2095 consensus_queue.cc:237] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.708506 1170 tablet_bootstrap.cc:492] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.712468 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.712785 1036 raft_consensus.cc:2466] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.713152 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:30.715029 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3315d3e9633e42ea8d7a6b5934cb0438" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.715116 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.715687 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31e332cdfedd4ff6b5026baa73e2caf3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:30.715734 891 tablet_service.cc:1468] Processing CreateTablet for tablet 10fef848b9b746c5a0de6a8c3ca74f7a (DEFAULT_TABLE table=default.Table_94f58de97c5946149d23d635eda4d555 [id=656d24e0a3ad4ac78696df5aec2f279e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.716025 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.716169 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10fef848b9b746c5a0de6a8c3ca74f7a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.716460 878 tablet_service.cc:1468] Processing CreateTablet for tablet e77bdf9df6eb4470a3d22e96da952718 (DEFAULT_TABLE table=default.Table_96c633b175ac41309951506efe002443 [id=ae92837e2ac94949aac8822f084c0a97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.716823 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e77bdf9df6eb4470a3d22e96da952718. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.717307 889 tablet_service.cc:1468] Processing CreateTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (DEFAULT_TABLE table=default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.717608 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.717787 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03a12cf719f34096bb82f5846bbe3b2d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.718305 1798 raft_consensus.cc:491] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.718437 2095 raft_consensus.cc:491] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.718490 768 raft_consensus.cc:3058] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.718427 1798 raft_consensus.cc:513] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.718530 2095 raft_consensus.cc:513] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.719066 1798 leader_election.cc:290] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:30.719345 2095 leader_election.cc:290] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.719856 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:30.720352 986 leader_election.cc:343] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.720835 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.721287 984 leader_election.cc:343] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.721385 984 leader_election.cc:304] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.721761 1798 raft_consensus.cc:2747] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.721771 883 tablet_service.cc:1468] Processing CreateTablet for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 (DEFAULT_TABLE table=default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.722168 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.723058 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6b8bbbcac06944469e13f14ab1d80402 (DEFAULT_TABLE table=default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.723368 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b8bbbcac06944469e13f14ab1d80402. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.724484 875 tablet_service.cc:1468] Processing CreateTablet for tablet 298a325f5e4645beac39eb59ddc0566d (DEFAULT_TABLE table=default.Table_44734742f0c948af9cce54466ff759c6 [id=11511883d6b046c7b155ff9687eb2c76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.724818 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 298a325f5e4645beac39eb59ddc0566d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.724851 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3ea2e2059a4425892fa3c0757e69e4c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.725533 848 leader_election.cc:343] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.725715 848 leader_election.cc:343] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.726020 848 consensus_peers.cc:489] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:30.726164 848 leader_election.cc:343] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.726404 848 leader_election.cc:343] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.726645 848 leader_election.cc:343] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.726722 848 leader_election.cc:304] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:30.727166 848 leader_election.cc:343] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.727779 2443 raft_consensus.cc:513] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.728121 2413 raft_consensus.cc:2747] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.730475 2444 raft_consensus.cc:513] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.731091 2444 leader_election.cc:290] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.731695 2379 ts_tablet_manager.cc:1918] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:30.731837 875 tablet_service.cc:1468] Processing CreateTablet for tablet 6edbc312fc4947a79e937dbc64d30b37 (DEFAULT_TABLE table=default.Table_56ae03b8d3eb4a58970a492506fcc6f1 [id=acf1b6990a594051ba94b43420cdeebc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.732013 2379 log.cc:1199] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/29c2b030fafd44ad9e648b2527a4ad07
I20250905 19:52:30.732657 2379 ts_tablet_manager.cc:1939] T 29c2b030fafd44ad9e648b2527a4ad07 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.734020 1170 tablet_bootstrap.cc:654] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.734921 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6edbc312fc4947a79e937dbc64d30b37. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.738880 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3315d3e9633e42ea8d7a6b5934cb0438" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:30.742662 848 leader_election.cc:343] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.744472 2443 leader_election.cc:290] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.745013 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:30.746059 848 leader_election.cc:343] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.746440 1171 tablet_bootstrap.cc:492] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.746594 1171 ts_tablet_manager.cc:1397] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.298s user 0.001s sys 0.011s
I20250905 19:52:30.747925 849 leader_election.cc:304] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.751843 2575 raft_consensus.cc:2802] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.752269 878 tablet_service.cc:1468] Processing CreateTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 (DEFAULT_TABLE table=default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.752858 889 tablet_service.cc:1468] Processing CreateTablet for tablet bb2b5ce70b784c13aa64053b2d1fe541 (DEFAULT_TABLE table=default.Table_57684ee2f11b44129e0b3d35029151fc [id=d636c310314a405e8d4f99e56d94f44c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.754212 1169 tablet_bootstrap.cc:492] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.754412 1169 ts_tablet_manager.cc:1397] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.160s user 0.008s sys 0.004s
I20250905 19:52:30.755249 1169 raft_consensus.cc:357] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.757030 1169 raft_consensus.cc:383] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.757159 1169 raft_consensus.cc:738] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.757453 1169 consensus_queue.cc:260] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.758119 1169 ts_tablet_manager.cc:1428] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:30.758236 1030 tablet_service.cc:1468] Processing CreateTablet for tablet bb2b5ce70b784c13aa64053b2d1fe541 (DEFAULT_TABLE table=default.Table_57684ee2f11b44129e0b3d35029151fc [id=d636c310314a405e8d4f99e56d94f44c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:30.766069 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 60) took 1428 ms. Trace:
I20250905 19:52:30.766182 2379 rpcz_store.cc:276] 0905 19:52:29.337222 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337620 (+ 398us) service_pool.cc:225] Handling call
0905 19:52:30.609533 (+1271913us) ts_tablet_manager.cc:1145] Deleting tablet 29c2b030fafd44ad9e648b2527a4ad07
0905 19:52:30.609545 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.682645 (+ 73100us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.731492 (+ 48847us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.766045 (+ 34553us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1263962}
I20250905 19:52:30.766484 2379 tablet_replica.cc:331] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.766650 2379 raft_consensus.cc:2241] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.766780 2379 raft_consensus.cc:2270] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.768922 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 29c2b030fafd44ad9e648b2527a4ad07 (table default.Table_f43b3348e3084812974970a576a6fd15 [id=bbdef6ec6956464f9cdd00b3fb506c97]) successfully deleted
I20250905 19:52:30.768034 2379 ts_tablet_manager.cc:1905] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.780157 2575 raft_consensus.cc:491] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.780297 2575 raft_consensus.cc:3058] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:30.787701 511 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.62:38687 timed out after 0.507s (SENT) [suppressed 2 similar messages]
I20250905 19:52:30.791096 2575 raft_consensus.cc:513] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.791774 2575 leader_election.cc:290] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.792460 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:30.793131 848 leader_election.cc:343] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.815253 2376 ts_tablet_manager.cc:1918] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.815402 2376 log.cc:1199] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:30.815758 2376 ts_tablet_manager.cc:1939] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.817919 1849 consensus_queue.cc:1035] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:30.823343 984 consensus_peers.cc:489] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.823940 1171 raft_consensus.cc:357] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.826260 1171 raft_consensus.cc:383] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.826508 1171 raft_consensus.cc:738] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.826915 1171 consensus_queue.cc:260] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.828290 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9f5d19f0216a4f069237e48c62ee7875. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.830051 2565 consensus_queue.cc:1035] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.831985 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2b5ce70b784c13aa64053b2d1fe541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.837807 1170 tablet_bootstrap.cc:492] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.837985 1170 ts_tablet_manager.cc:1397] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.131s user 0.005s sys 0.006s
I20250905 19:52:30.839933 1170 raft_consensus.cc:357] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.840369 1170 raft_consensus.cc:383] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.840747 1170 raft_consensus.cc:738] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.841171 1170 consensus_queue.cc:260] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.842859 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 61) took 1576 ms. Trace:
I20250905 19:52:30.842967 2376 rpcz_store.cc:276] 0905 19:52:29.266336 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.266417 (+ 81us) service_pool.cc:225] Handling call
0905 19:52:30.437869 (+1171452us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:30.437880 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.786605 (+348725us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.815135 (+ 28530us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.842841 (+ 27706us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1171304}
I20250905 19:52:30.843470 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff (table default.Table_1c6571236cd7434d98aec149d06dc6b8 [id=6709b02c192c4ff88715b1f3b886b222]) successfully deleted
I20250905 19:52:30.843274 2376 tablet_replica.cc:331] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.844062 2376 raft_consensus.cc:2241] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.844561 2376 raft_consensus.cc:2270] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.844791 1798 raft_consensus.cc:491] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.844945 1798 raft_consensus.cc:513] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.845544 1798 leader_election.cc:290] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:30.845815 2376 ts_tablet_manager.cc:1905] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.846303 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1b4989f17454aa58c05727b797c2009" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.846468 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1b4989f17454aa58c05727b797c2009" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.846887 986 leader_election.cc:343] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.847134 984 leader_election.cc:343] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.847232 984 leader_election.cc:304] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.847615 1798 raft_consensus.cc:2747] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.847707 1170 ts_tablet_manager.cc:1428] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20250905 19:52:30.851290 1798 raft_consensus.cc:491] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.851054 1170 tablet_bootstrap.cc:492] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.851404 1798 raft_consensus.cc:513] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.852013 1798 leader_election.cc:290] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:30.852643 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6221d7d058944bdb9aa730791cf2417b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.853021 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6221d7d058944bdb9aa730791cf2417b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.853245 986 leader_election.cc:343] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:30.853438 984 leader_election.cc:343] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.853526 984 leader_election.cc:304] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.853863 1798 raft_consensus.cc:2747] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.854722 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:30.857586 2379 ts_tablet_manager.cc:1918] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:30.859297 2379 log.cc:1199] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a8b6823580cb4040b49ed0a49b748965
I20250905 19:52:30.859719 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:30.865235 1171 ts_tablet_manager.cc:1428] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.119s user 0.000s sys 0.002s
I20250905 19:52:30.865595 1171 tablet_bootstrap.cc:492] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.867077 1171 tablet_bootstrap.cc:654] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.881552 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3315d3e9633e42ea8d7a6b5934cb0438" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.881738 1037 raft_consensus.cc:3058] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.881790 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3ea2e2059a4425892fa3c0757e69e4c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.881968 1036 raft_consensus.cc:2466] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.882228 2575 consensus_queue.cc:1035] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.882581 2575 consensus_queue.cc:1035] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.883296 849 leader_election.cc:304] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.884032 1039 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.884083 2577 raft_consensus.cc:2802] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.884155 2577 raft_consensus.cc:491] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.884169 1039 raft_consensus.cc:3058] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.884214 2577 raft_consensus.cc:3058] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.886637 1039 raft_consensus.cc:2466] T 8092952d755848ccaba3a2ad94b66dad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.886925 2577 raft_consensus.cc:513] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.887051 849 leader_election.cc:304] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.887403 2575 raft_consensus.cc:2802] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.887490 2577 leader_election.cc:290] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.887511 2575 raft_consensus.cc:695] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.887602 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.887749 1051 raft_consensus.cc:3058] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.887734 2575 consensus_queue.cc:237] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.890390 1051 raft_consensus.cc:2466] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.892545 1037 raft_consensus.cc:2466] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.893136 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb2b5ce70b784c13aa64053b2d1fe541. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:30.893039 1798 consensus_queue.cc:1035] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.894403 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:30.894958 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:30.895949 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:30.896610 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:30.896970 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
W20250905 19:52:30.892174 711 consensus_peers.cc:489] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.864630 2379 ts_tablet_manager.cc:1939] T a8b6823580cb4040b49ed0a49b748965 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.898418 849 leader_election.cc:304] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:30.898910 849 consensus_peers.cc:489] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.899087 849 leader_election.cc:304] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.899066 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3ea2e2059a4425892fa3c0757e69e4c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:30.899580 848 consensus_peers.cc:489] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.899653 2007 raft_consensus.cc:2802] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.899708 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b3ea2e2059a4425892fa3c0757e69e4c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.899760 2007 raft_consensus.cc:695] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.899847 1037 raft_consensus.cc:3058] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.900095 2007 consensus_queue.cc:237] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:30.900219 848 leader_election.cc:343] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.901507 2108 raft_consensus.cc:491] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.901625 2108 raft_consensus.cc:513] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.902096 2108 leader_election.cc:290] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.899588 2095 raft_consensus.cc:2802] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.902508 2095 raft_consensus.cc:695] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.902679 774 raft_consensus.cc:3058] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.902707 2095 consensus_queue.cc:237] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.904178 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.904350 1044 raft_consensus.cc:2466] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.905128 1037 raft_consensus.cc:2466] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.907028 1849 raft_consensus.cc:491] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.907183 1849 raft_consensus.cc:513] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.907462 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:30.907776 1849 leader_election.cc:290] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:30.908736 848 leader_election.cc:343] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.909184 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f1a8e7059b348cba34069ef49bf1aad" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:30.909730 849 leader_election.cc:304] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.910447 2108 raft_consensus.cc:2802] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.910524 2108 raft_consensus.cc:491] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.910584 2108 raft_consensus.cc:3058] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.909291 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1f1a8e7059b348cba34069ef49bf1aad" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.912866 1170 tablet_bootstrap.cc:654] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.913471 2108 raft_consensus.cc:513] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.913830 2568 raft_consensus.cc:491] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:30.913936 2568 raft_consensus.cc:513] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.914001 2108 leader_election.cc:290] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.914366 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:30.914431 2568 leader_election.cc:290] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.914646 712 leader_election.cc:304] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:30.914943 848 leader_election.cc:343] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.914952 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:30.915369 848 leader_election.cc:343] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.916349 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.916522 1044 raft_consensus.cc:3058] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.918816 2100 raft_consensus.cc:2747] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:30.919126 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:30.919286 1046 raft_consensus.cc:2466] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:30.919929 1044 raft_consensus.cc:2466] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.920778 849 leader_election.cc:304] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.920809 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 55) took 1584 ms. Trace:
I20250905 19:52:30.920969 2379 rpcz_store.cc:276] 0905 19:52:29.336680 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337837 (+ 1157us) service_pool.cc:225] Handling call
0905 19:52:30.766443 (+1428606us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:30.766452 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.786103 (+ 19651us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.855926 (+ 69823us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.920740 (+ 64814us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1421152}
I20250905 19:52:30.921746 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a8b6823580cb4040b49ed0a49b748965 (table default.Table_aa894f644dbe444bbe04d396b9ceb056 [id=4172afda1ef0438cb4f4695bb8340e66]) successfully deleted
I20250905 19:52:30.921792 2568 raft_consensus.cc:2802] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.921780 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 59) took 1584 ms. Trace:
I20250905 19:52:30.921907 2568 raft_consensus.cc:695] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.921980 2379 rpcz_store.cc:276] 0905 19:52:29.337107 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337593 (+ 486us) service_pool.cc:225] Handling call
0905 19:52:30.921550 (+1583957us) ts_tablet_manager.cc:1145] Deleting tablet 262bcd1cfc994bb58e999b141b9bc7ea
0905 19:52:30.921561 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.921704 (+ 143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1566223}
I20250905 19:52:30.922139 2568 consensus_queue.cc:237] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:30.922719 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 262bcd1cfc994bb58e999b141b9bc7ea because the tablet was not found. No further retry: Not found: Tablet not found: 262bcd1cfc994bb58e999b141b9bc7ea
I20250905 19:52:30.922662 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 57) took 1585 ms. Trace:
I20250905 19:52:30.922849 2379 rpcz_store.cc:276] 0905 19:52:29.336883 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337535 (+ 652us) service_pool.cc:225] Handling call
0905 19:52:30.922442 (+1584907us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:30.922451 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.922592 (+ 141us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567043}
I20250905 19:52:30.923313 2379 tablet_replica.cc:331] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:30.923432 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6ebab2e3698f4f60a770bde9cc019600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ebab2e3698f4f60a770bde9cc019600 already in progress: creating tablet
I20250905 19:52:30.923708 2379 raft_consensus.cc:2241] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.921804 849 leader_election.cc:304] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.924021 2379 raft_consensus.cc:2270] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.924718 2443 raft_consensus.cc:2802] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:30.924809 2443 raft_consensus.cc:491] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:30.924880 2443 raft_consensus.cc:3058] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.897385 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:30.926434 2376 ts_tablet_manager.cc:1918] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:30.927742 2376 log.cc:1199] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/05ae83eea5f74e9187713e8b82c9a23b
I20250905 19:52:30.927788 2443 raft_consensus.cc:513] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.928232 1169 tablet_bootstrap.cc:492] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:30.928359 2376 ts_tablet_manager.cc:1939] T 05ae83eea5f74e9187713e8b82c9a23b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:30.929189 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 60) took 1663 ms. Trace:
I20250905 19:52:30.929929 1169 tablet_bootstrap.cc:654] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.930039 2376 rpcz_store.cc:276] 0905 19:52:29.266206 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.266591 (+ 385us) service_pool.cc:225] Handling call
0905 19:52:30.843231 (+1576640us) ts_tablet_manager.cc:1145] Deleting tablet 05ae83eea5f74e9187713e8b82c9a23b
0905 19:52:30.843239 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.850757 (+ 7518us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.926326 (+ 75569us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.929168 (+ 2842us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1576466}
I20250905 19:52:30.930581 1170 tablet_bootstrap.cc:492] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:30.931061 849 leader_election.cc:304] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:30.932585 2007 raft_consensus.cc:2802] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.932979 2007 raft_consensus.cc:695] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.932777 1170 ts_tablet_manager.cc:1397] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.082s user 0.003s sys 0.003s
I20250905 19:52:30.933305 2007 consensus_queue.cc:237] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.932724 2379 ts_tablet_manager.cc:1905] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.933741 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 05ae83eea5f74e9187713e8b82c9a23b (table default.Table_ab2b806012994ad38e6f7a3171f00777 [id=29f567c56184458c96d24eb78f402137]) successfully deleted
I20250905 19:52:30.933311 2443 leader_election.cc:290] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:30.936335 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:30.943329 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:30.943275 2443 consensus_queue.cc:1035] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:30.943508 1037 raft_consensus.cc:3058] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:30.943740 1171 tablet_bootstrap.cc:492] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.943853 1171 ts_tablet_manager.cc:1397] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.078s user 0.000s sys 0.002s
I20250905 19:52:30.944490 1171 raft_consensus.cc:357] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.944625 1171 raft_consensus.cc:383] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.944676 1171 raft_consensus.cc:738] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.944859 1171 consensus_queue.cc:260] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.945307 1171 ts_tablet_manager.cc:1428] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 19:52:30.945547 1171 tablet_bootstrap.cc:492] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:30.946151 1037 raft_consensus.cc:2466] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:30.947012 1171 tablet_bootstrap.cc:654] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.947427 849 leader_election.cc:304] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:30.947682 2422 raft_consensus.cc:2802] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:30.947793 2422 raft_consensus.cc:695] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:30.948011 2422 consensus_queue.cc:237] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.948333 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:30.948920 848 leader_election.cc:343] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:30.960987 1171 tablet_bootstrap.cc:492] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:30.961126 1171 ts_tablet_manager.cc:1397] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.003s
I20250905 19:52:30.961772 1171 raft_consensus.cc:357] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.962087 1171 raft_consensus.cc:383] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.962167 1171 raft_consensus.cc:738] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.962486 1171 consensus_queue.cc:260] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.962992 1171 ts_tablet_manager.cc:1428] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:30.963272 1171 tablet_bootstrap.cc:492] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:30.965571 848 consensus_peers.cc:489] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:30.966737 1171 tablet_bootstrap.cc:654] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.960170 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:30.969522 1170 raft_consensus.cc:357] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.969584 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:30.969748 1170 raft_consensus.cc:383] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.969818 1170 raft_consensus.cc:738] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.970093 1170 consensus_queue.cc:260] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.970975 1170 ts_tablet_manager.cc:1428] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.037s user 0.000s sys 0.002s
I20250905 19:52:30.971319 1170 tablet_bootstrap.cc:492] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:30.972972 1170 tablet_bootstrap.cc:654] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:30.974344 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:30.933593 2376 tablet_replica.cc:331] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:30.978509 2376 raft_consensus.cc:2241] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.978790 2376 raft_consensus.cc:2270] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.979678 2376 ts_tablet_manager.cc:1905] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.976161 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:30.984117 2379 ts_tablet_manager.cc:1918] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:30.984289 2379 log.cc:1199] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:30.984666 2379 ts_tablet_manager.cc:1939] T d480531ba94f4b0f9fe40e88fe91512e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:30.985039 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 56) took 1648 ms. Trace:
I20250905 19:52:30.985150 2379 rpcz_store.cc:276] 0905 19:52:29.336781 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337808 (+ 1027us) service_pool.cc:225] Handling call
0905 19:52:30.923201 (+1585393us) ts_tablet_manager.cc:1145] Deleting tablet d480531ba94f4b0f9fe40e88fe91512e
0905 19:52:30.923239 (+ 38us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:30.967758 (+ 44519us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.983953 (+ 16195us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:30.984996 (+ 1043us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1567074}
I20250905 19:52:30.985543 2379 tablet_replica.cc:331] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:30.985087 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:30.985769 2379 raft_consensus.cc:2241] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:30.985770 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d480531ba94f4b0f9fe40e88fe91512e (table default.Table_c7b1f5de2f2f4657971e14388d57ef5a [id=d92e1dc4017f4f8cbfdeeb40900f136c]) successfully deleted
I20250905 19:52:30.985962 2379 raft_consensus.cc:2270] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:30.994314 2379 ts_tablet_manager.cc:1905] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:30.994774 1169 tablet_bootstrap.cc:492] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:30.995018 1169 ts_tablet_manager.cc:1397] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.008s
I20250905 19:52:30.996091 1169 raft_consensus.cc:357] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:30.996435 1169 raft_consensus.cc:383] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:30.996644 1169 raft_consensus.cc:738] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:30.997124 1169 consensus_queue.cc:260] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:31.002843 848 consensus_peers.cc:489] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.007283 2100 raft_consensus.cc:491] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.007464 2100 raft_consensus.cc:513] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.008112 2100 leader_election.cc:290] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:31.009230 711 leader_election.cc:343] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:31.009035 1169 ts_tablet_manager.cc:1428] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
W20250905 19:52:31.012328 984 consensus_peers.cc:489] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.012605 1169 tablet_bootstrap.cc:492] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.012980 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03a12cf719f34096bb82f5846bbe3b2d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:31.013170 914 raft_consensus.cc:2466] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:31.013605 712 leader_election.cc:304] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:31.014062 2100 raft_consensus.cc:2802] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.014171 2100 raft_consensus.cc:491] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.014253 2100 raft_consensus.cc:3058] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.016884 2100 raft_consensus.cc:513] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.017844 2100 leader_election.cc:290] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.018345 2379 ts_tablet_manager.cc:1918] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.018465 2379 log.cc:1199] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/42f3c1efb94e446fb0128b6b412821e2
I20250905 19:52:31.020319 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03a12cf719f34096bb82f5846bbe3b2d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:31.020479 914 raft_consensus.cc:3058] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.023064 914 raft_consensus.cc:2466] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:31.025038 712 leader_election.cc:304] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:31.026052 2100 raft_consensus.cc:2802] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.026312 2100 raft_consensus.cc:695] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:31.026754 2100 consensus_queue.cc:237] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.030810 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:31.031693 2379 ts_tablet_manager.cc:1939] T 42f3c1efb94e446fb0128b6b412821e2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.031292 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:31.032902 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:31.033337 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:31.033288 2376 ts_tablet_manager.cc:1918] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.033727 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:31.033927 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]). Expected version 5 got 0
I20250905 19:52:31.033866 2376 log.cc:1199] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:31.034598 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:31.034905 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:31.035224 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
W20250905 19:52:31.035461 711 leader_election.cc:343] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:31.035543 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:31.035519 2376 ts_tablet_manager.cc:1939] T 302053e1611b4e22a9a2e7b60ae68f27 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:31.035820 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]). Expected version 1 got 0
I20250905 19:52:31.036057 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:31.036194 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:31.036396 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:31.036832 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:31.037086 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:31.037492 1169 tablet_bootstrap.cc:654] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.037705 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:31.038093 1170 tablet_bootstrap.cc:492] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.038262 1170 ts_tablet_manager.cc:1397] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.067s user 0.003s sys 0.000s
I20250905 19:52:31.038411 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:31.039022 1170 raft_consensus.cc:357] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.039253 1170 raft_consensus.cc:383] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.039340 1170 raft_consensus.cc:738] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.039587 1170 consensus_queue.cc:260] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.039933 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 64) took 1773 ms. Trace:
I20250905 19:52:31.040541 2496 raft_consensus.cc:491] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.040704 2496 raft_consensus.cc:513] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.040408 2376 rpcz_store.cc:276] 0905 19:52:29.266669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.266793 (+ 124us) service_pool.cc:225] Handling call
0905 19:52:30.933541 (+1666748us) ts_tablet_manager.cc:1145] Deleting tablet 302053e1611b4e22a9a2e7b60ae68f27
0905 19:52:30.933553 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.020175 (+ 86622us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.032950 (+ 12775us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.039904 (+ 6954us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1666574}
I20250905 19:52:31.041481 2496 leader_election.cc:290] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.041540 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 302053e1611b4e22a9a2e7b60ae68f27 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:31.042513 2376 tablet_replica.cc:331] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.042730 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24bcd21f19ba465aa550102e0fccb674" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.043322 986 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.043532 2376 raft_consensus.cc:2241] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.044087 2376 raft_consensus.cc:2270] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.044238 1171 tablet_bootstrap.cc:492] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.044425 1170 ts_tablet_manager.cc:1428] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20250905 19:52:31.044641 1171 ts_tablet_manager.cc:1397] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.081s user 0.005s sys 0.002s
I20250905 19:52:31.045763 1171 raft_consensus.cc:357] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.046072 1171 raft_consensus.cc:383] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.046214 1171 raft_consensus.cc:738] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.046633 1171 consensus_queue.cc:260] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.047628 1171 ts_tablet_manager.cc:1428] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:31.049082 1171 tablet_bootstrap.cc:492] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.051172 2376 ts_tablet_manager.cc:1905] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:31.053654 848 consensus_peers.cc:489] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.053490 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24bcd21f19ba465aa550102e0fccb674" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.054319 984 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.054436 984 leader_election.cc:304] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.055358 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 54) took 1718 ms. Trace:
I20250905 19:52:31.055516 2379 rpcz_store.cc:276] 0905 19:52:29.336495 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.337923 (+ 1428us) service_pool.cc:225] Handling call
0905 19:52:30.985469 (+1647546us) ts_tablet_manager.cc:1145] Deleting tablet 42f3c1efb94e446fb0128b6b412821e2
0905 19:52:30.985480 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.001251 (+ 15771us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.018216 (+ 16965us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.055310 (+ 37094us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1621358}
I20250905 19:52:31.056030 2379 tablet_replica.cc:331] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:31.056344 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 42f3c1efb94e446fb0128b6b412821e2 (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]) successfully deleted
I20250905 19:52:31.056488 2379 raft_consensus.cc:2241] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:31.056053 1170 tablet_bootstrap.cc:492] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.056892 2379 raft_consensus.cc:2270] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.057492 2379 ts_tablet_manager.cc:1905] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.058358 1170 tablet_bootstrap.cc:654] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.059149 1171 tablet_bootstrap.cc:654] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.062484 1169 tablet_bootstrap.cc:492] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.063086 1169 ts_tablet_manager.cc:1397] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.001s
I20250905 19:52:31.063745 1798 raft_consensus.cc:491] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.063916 1798 raft_consensus.cc:513] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.065006 1798 leader_election.cc:290] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.065804 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db56577a26e407b97fc33158c2c466c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.066510 986 leader_election.cc:343] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.070284 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db56577a26e407b97fc33158c2c466c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.070020 1169 raft_consensus.cc:357] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:31.071126 984 leader_election.cc:343] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.071444 984 leader_election.cc:304] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.071125 1169 raft_consensus.cc:383] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.071542 1169 raft_consensus.cc:738] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.072021 2609 raft_consensus.cc:2747] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.072249 1798 raft_consensus.cc:2747] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.072928 1169 consensus_queue.cc:260] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.075794 2568 raft_consensus.cc:491] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.075991 1169 ts_tablet_manager.cc:1428] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250905 19:52:31.076009 2568 raft_consensus.cc:513] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.077867 1169 tablet_bootstrap.cc:492] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.081341 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3a50526bb604a218649b24cbf65a65e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.082084 848 leader_election.cc:343] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.087885 712 consensus_peers.cc:489] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.088335 1169 tablet_bootstrap.cc:654] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.089315 2376 ts_tablet_manager.cc:1918] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.090015 2565 consensus_queue.cc:1035] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.089987 2376 log.cc:1199] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/1f1a8e7059b348cba34069ef49bf1aad
I20250905 19:52:31.092460 2376 ts_tablet_manager.cc:1939] T 1f1a8e7059b348cba34069ef49bf1aad P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:31.092913 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3a50526bb604a218649b24cbf65a65e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.093086 1042 raft_consensus.cc:2466] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:31.093538 849 leader_election.cc:304] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.093966 2565 raft_consensus.cc:2802] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.094094 2565 raft_consensus.cc:491] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.094204 2565 raft_consensus.cc:3058] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.095412 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 59) took 1829 ms. Trace:
I20250905 19:52:31.095646 2376 rpcz_store.cc:276] 0905 19:52:29.266091 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.266986 (+ 895us) service_pool.cc:225] Handling call
0905 19:52:31.042256 (+1775270us) ts_tablet_manager.cc:1145] Deleting tablet 1f1a8e7059b348cba34069ef49bf1aad
0905 19:52:31.042270 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.071642 (+ 29372us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.088481 (+ 16839us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.095301 (+ 6820us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1775005}
I20250905 19:52:31.096722 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 1f1a8e7059b348cba34069ef49bf1aad (table default.Table_7186efc2caef4940ade686830c963ba1 [id=0a06a88771a94e60b5bb57fa7f7d8215]) successfully deleted
I20250905 19:52:31.097395 2565 raft_consensus.cc:513] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.097787 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 57) took 1831 ms. Trace:
I20250905 19:52:31.098793 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3a50526bb604a218649b24cbf65a65e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:31.098206 2376 rpcz_store.cc:276] 0905 19:52:29.265842 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267172 (+ 1330us) service_pool.cc:225] Handling call
0905 19:52:31.097235 (+1830063us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:31.097334 (+ 99us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.097763 (+ 429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1829900}
W20250905 19:52:31.099423 848 leader_election.cc:343] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.099721 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 14d5eb8bec074d2c8766fe7b159031e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14d5eb8bec074d2c8766fe7b159031e5 already in progress: creating tablet
I20250905 19:52:31.099916 768 raft_consensus.cc:3058] T de19c897d3314fecb17cedb3370eed93 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.100493 2376 tablet_replica.cc:331] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.101161 2376 raft_consensus.cc:2241] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.101778 2376 raft_consensus.cc:2270] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.104135 1171 tablet_bootstrap.cc:492] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.104300 1171 ts_tablet_manager.cc:1397] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.056s user 0.005s sys 0.001s
I20250905 19:52:31.104349 2568 leader_election.cc:290] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.105078 1171 raft_consensus.cc:357] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.105285 1171 raft_consensus.cc:383] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.105376 1171 raft_consensus.cc:738] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.105600 1171 consensus_queue.cc:260] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.106125 1171 ts_tablet_manager.cc:1428] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:31.106866 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:31.108300 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a3a50526bb604a218649b24cbf65a65e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:31.108489 1042 raft_consensus.cc:3058] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.109495 2565 leader_election.cc:290] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.111256 1042 raft_consensus.cc:2466] T a3a50526bb604a218649b24cbf65a65e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:31.111760 849 leader_election.cc:304] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.112027 2496 raft_consensus.cc:491] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.112159 2496 raft_consensus.cc:513] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.113118 2496 leader_election.cc:290] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.113198 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0c478efe044ad0867e47f49374a66b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.113737 986 leader_election.cc:343] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.115113 2565 raft_consensus.cc:2802] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.115854 1171 tablet_bootstrap.cc:492] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.116205 2379 ts_tablet_manager.cc:1918] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.116353 2379 log.cc:1199] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:31.116787 2379 ts_tablet_manager.cc:1939] T 4d2ed1bc09c94526bc9662f2d4d21f8f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.118443 1171 tablet_bootstrap.cc:654] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.121614 769 raft_consensus.cc:3058] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.138612 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 63) took 1799 ms. Trace:
I20250905 19:52:31.138744 2379 rpcz_store.cc:276] 0905 19:52:29.339074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.339171 (+ 97us) service_pool.cc:225] Handling call
0905 19:52:31.055925 (+1716754us) ts_tablet_manager.cc:1145] Deleting tablet 4d2ed1bc09c94526bc9662f2d4d21f8f
0905 19:52:31.055934 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.073386 (+ 17452us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.116046 (+ 42660us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.138562 (+ 22516us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1689829}
I20250905 19:52:31.139106 2379 tablet_replica.cc:331] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:31.139240 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4d2ed1bc09c94526bc9662f2d4d21f8f (table default.Table_315b3dd493ac42fea5c100b3e58d9533 [id=70f8481033284889b47c81173fb2ef5c]) successfully deleted
I20250905 19:52:31.139310 2379 raft_consensus.cc:2241] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.139473 2379 raft_consensus.cc:2270] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.140028 2379 ts_tablet_manager.cc:1905] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.141459 1171 tablet_bootstrap.cc:492] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.141662 1171 ts_tablet_manager.cc:1397] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250905 19:52:31.142017 1170 tablet_bootstrap.cc:492] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.142210 1170 ts_tablet_manager.cc:1397] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.086s user 0.007s sys 0.000s
I20250905 19:52:31.143014 1170 raft_consensus.cc:357] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.143317 1170 raft_consensus.cc:383] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.143443 1170 raft_consensus.cc:738] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.143780 1170 consensus_queue.cc:260] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.144593 2376 ts_tablet_manager.cc:1905] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.144618 1170 ts_tablet_manager.cc:1428] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:31.145318 1169 tablet_bootstrap.cc:492] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.145515 1169 ts_tablet_manager.cc:1397] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.068s user 0.005s sys 0.013s
I20250905 19:52:31.146719 1169 raft_consensus.cc:357] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.146988 1169 raft_consensus.cc:383] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.147070 1169 raft_consensus.cc:738] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.147408 1169 consensus_queue.cc:260] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.148090 2565 raft_consensus.cc:695] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:31.148488 1169 ts_tablet_manager.cc:1428] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:31.148415 2565 consensus_queue.cc:237] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.148962 1169 tablet_bootstrap.cc:492] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.151885 1171 raft_consensus.cc:357] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:31.152495 848 consensus_peers.cc:489] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.152571 1171 raft_consensus.cc:383] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.152657 1171 raft_consensus.cc:738] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.152966 1171 consensus_queue.cc:260] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.154160 1171 ts_tablet_manager.cc:1428] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250905 19:52:31.154592 1171 tablet_bootstrap.cc:492] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.155503 1170 tablet_bootstrap.cc:492] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.147508 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0c478efe044ad0867e47f49374a66b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.156611 984 leader_election.cc:343] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.156777 984 leader_election.cc:304] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.157011 2496 raft_consensus.cc:2747] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.159809 1171 tablet_bootstrap.cc:654] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.160271 1169 tablet_bootstrap.cc:654] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.160943 2376 ts_tablet_manager.cc:1918] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.161043 2376 log.cc:1199] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a5f076d9917f40d899d7422ad974fe0f
W20250905 19:52:31.169637 848 consensus_peers.cc:489] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.188186 1169 tablet_bootstrap.cc:492] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.188728 1169 ts_tablet_manager.cc:1397] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.040s user 0.008s sys 0.001s
I20250905 19:52:31.191994 1169 raft_consensus.cc:357] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.193032 1170 tablet_bootstrap.cc:654] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.193760 1171 tablet_bootstrap.cc:492] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.193996 1171 ts_tablet_manager.cc:1397] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.039s user 0.003s sys 0.006s
I20250905 19:52:31.195334 1171 raft_consensus.cc:357] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.195600 1171 raft_consensus.cc:383] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.195686 1171 raft_consensus.cc:738] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.196000 2376 ts_tablet_manager.cc:1939] T a5f076d9917f40d899d7422ad974fe0f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:31.196015 1171 consensus_queue.cc:260] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.196468 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 69) took 1929 ms. Trace:
I20250905 19:52:31.196596 2376 rpcz_store.cc:276] 0905 19:52:29.267288 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267372 (+ 84us) service_pool.cc:225] Handling call
0905 19:52:31.100266 (+1832894us) ts_tablet_manager.cc:1145] Deleting tablet a5f076d9917f40d899d7422ad974fe0f
0905 19:52:31.100367 (+ 101us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.157841 (+ 57474us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.160831 (+ 2990us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.196452 (+ 35621us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1832689}
I20250905 19:52:31.197115 2376 tablet_replica.cc:331] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.197080 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a5f076d9917f40d899d7422ad974fe0f (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:31.196858 1169 raft_consensus.cc:383] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.197480 1171 ts_tablet_manager.cc:1428] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:31.197651 1169 raft_consensus.cc:738] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.198762 1171 tablet_bootstrap.cc:492] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.198813 1169 consensus_queue.cc:260] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.201732 1169 ts_tablet_manager.cc:1428] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:31.203296 2376 raft_consensus.cc:2241] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:31.203835 1169 tablet_bootstrap.cc:492] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.205437 2376 raft_consensus.cc:2270] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.210043 2376 ts_tablet_manager.cc:1905] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.211074 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:31.212069 2496 consensus_queue.cc:1035] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.217805 2496 raft_consensus.cc:491] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.217963 2496 raft_consensus.cc:513] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.218595 2496 leader_election.cc:290] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.219250 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.219791 986 leader_election.cc:343] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.220484 1171 tablet_bootstrap.cc:654] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.225311 1169 tablet_bootstrap.cc:654] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.229593 2574 consensus_queue.cc:1035] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.235529 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.237597 984 leader_election.cc:343] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.237742 984 leader_election.cc:304] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.238236 2496 raft_consensus.cc:2747] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.242641 1169 tablet_bootstrap.cc:492] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.244139 1169 ts_tablet_manager.cc:1397] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.001s
I20250905 19:52:31.248414 1171 tablet_bootstrap.cc:492] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.249488 1171 ts_tablet_manager.cc:1397] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20250905 19:52:31.252295 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:31.253803 1169 raft_consensus.cc:357] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.255497 1169 raft_consensus.cc:383] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.256016 1169 raft_consensus.cc:738] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.257833 1169 consensus_queue.cc:260] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.258023 1171 raft_consensus.cc:357] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.259305 1171 raft_consensus.cc:383] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.259902 1171 raft_consensus.cc:738] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.261848 1170 tablet_bootstrap.cc:492] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.262033 1170 ts_tablet_manager.cc:1397] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.107s user 0.003s sys 0.000s
I20250905 19:52:31.262017 1169 ts_tablet_manager.cc:1428] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.017s user 0.000s sys 0.002s
I20250905 19:52:31.263487 1170 raft_consensus.cc:357] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.263697 1170 raft_consensus.cc:383] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.263779 1170 raft_consensus.cc:738] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.264036 1170 consensus_queue.cc:260] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.265760 1171 consensus_queue.cc:260] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.267899 1169 tablet_bootstrap.cc:492] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.269786 1170 ts_tablet_manager.cc:1428] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:31.270092 1170 tablet_bootstrap.cc:492] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.270227 1171 ts_tablet_manager.cc:1428] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20250905 19:52:31.271744 1170 tablet_bootstrap.cc:654] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.272883 1171 tablet_bootstrap.cc:492] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.283083 2376 ts_tablet_manager.cc:1918] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.284407 2376 log.cc:1199] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/53ac46ce05694bc2b53e195638be9e23
I20250905 19:52:31.284758 2496 raft_consensus.cc:491] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.284830 1169 tablet_bootstrap.cc:654] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.284890 2496 raft_consensus.cc:513] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.285959 2496 leader_election.cc:290] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.286713 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:31.286855 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.287599 986 leader_election.cc:343] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.289302 2379 ts_tablet_manager.cc:1918] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:31.289448 2379 log.cc:1199] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/60a41dbd8782460cb0d3c5f8bd59354f
W20250905 19:52:31.289563 984 leader_election.cc:343] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.289667 984 leader_election.cc:304] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.289819 2379 ts_tablet_manager.cc:1939] T 60a41dbd8782460cb0d3c5f8bd59354f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.289687 2376 ts_tablet_manager.cc:1939] T 53ac46ce05694bc2b53e195638be9e23 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:31.289757 1171 tablet_bootstrap.cc:654] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.290086 2496 raft_consensus.cc:2747] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.290995 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 64) took 1950 ms. Trace:
I20250905 19:52:31.291201 2379 rpcz_store.cc:276] 0905 19:52:29.364705 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.364845 (+ 140us) service_pool.cc:225] Handling call
0905 19:52:31.139031 (+1774186us) ts_tablet_manager.cc:1145] Deleting tablet 60a41dbd8782460cb0d3c5f8bd59354f
0905 19:52:31.139041 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.143330 (+ 4289us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.289168 (+145838us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.290893 (+ 1725us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1771017}
I20250905 19:52:31.291890 2379 tablet_replica.cc:331] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:31.291955 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 60a41dbd8782460cb0d3c5f8bd59354f (table default.Table_cddec42a8c1c415ba27927b52d055a08 [id=c60ff09c1e55439cbac9002d52e8d546]) successfully deleted
I20250905 19:52:31.292480 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:31.292869 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:31.293550 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:31.294011 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:31.294385 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:31.297559 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 46) took 2033 ms. Trace:
W20250905 19:52:31.299057 712 consensus_peers.cc:489] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.298139 2376 rpcz_store.cc:276] 0905 19:52:29.264547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267566 (+ 3019us) service_pool.cc:225] Handling call
0905 19:52:31.197070 (+1929504us) ts_tablet_manager.cc:1145] Deleting tablet 53ac46ce05694bc2b53e195638be9e23
0905 19:52:31.197080 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.248867 (+ 51787us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.281509 (+ 32642us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.297315 (+ 15806us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4789,"tablet-delete.queue_time_us":1929218}
I20250905 19:52:31.303117 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 53ac46ce05694bc2b53e195638be9e23 (table default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]) successfully deleted
I20250905 19:52:31.295725 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:31.303836 2376 tablet_replica.cc:331] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.304816 2376 raft_consensus.cc:2241] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.305408 2376 raft_consensus.cc:2270] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.305868 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:31.306104 2379 raft_consensus.cc:2241] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.308200 2379 raft_consensus.cc:2270] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:31.311167 511 leader_election.cc:336] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Timed out: RequestConsensusVote RPC to 127.0.104.62:38687 timed out after 0.507s (SENT)
I20250905 19:52:31.311316 511 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:31.314352 2376 ts_tablet_manager.cc:1905] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.318331 1169 tablet_bootstrap.cc:492] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.318611 1169 ts_tablet_manager.cc:1397] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.051s user 0.003s sys 0.004s
I20250905 19:52:31.318904 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:31.322270 2634 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:52:31.323884 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]). Expected version 3 got 0
I20250905 19:52:31.324470 1169 raft_consensus.cc:357] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.325290 1169 raft_consensus.cc:383] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.325582 1169 raft_consensus.cc:738] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.325497 2379 ts_tablet_manager.cc:1905] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.326506 1169 consensus_queue.cc:260] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.329707 1169 ts_tablet_manager.cc:1428] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.003s sys 0.000s
I20250905 19:52:31.331066 769 raft_consensus.cc:3058] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.331346 1169 tablet_bootstrap.cc:492] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:31.334602 1479 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e10b4e82c4bc429b9a42db39c03a5267 consensus state has no leader
I20250905 19:52:31.336424 774 raft_consensus.cc:3058] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.344477 2574 consensus_queue.cc:1035] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.347662 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
W20250905 19:52:31.350880 986 consensus_peers.cc:489] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ab4f69c45295422f85b310aa55220e9b. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.356808 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:31.361708 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:31.364887 1171 tablet_bootstrap.cc:492] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.365041 1171 ts_tablet_manager.cc:1397] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.093s user 0.007s sys 0.004s
I20250905 19:52:31.365734 1171 raft_consensus.cc:357] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.365911 1171 raft_consensus.cc:383] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.365967 1171 raft_consensus.cc:738] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.368916 2376 ts_tablet_manager.cc:1918] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.369277 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:31.370925 1171 consensus_queue.cc:260] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.372845 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]). Expected version 2 got 0
I20250905 19:52:31.377056 1171 ts_tablet_manager.cc:1428] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.012s user 0.001s sys 0.001s
I20250905 19:52:31.378094 2376 log.cc:1199] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:31.378484 1169 tablet_bootstrap.cc:654] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:31.379169 848 consensus_peers.cc:489] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.380532 1171 tablet_bootstrap.cc:492] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:31.381717 986 consensus_peers.cc:489] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1f1a8e7059b348cba34069ef49bf1aad. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.382863 2376 ts_tablet_manager.cc:1939] T b13d282e1dc54d1faa2882c9b39d17cb P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:31.383322 848 consensus_peers.cc:489] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.386119 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:31.389423 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 65) took 2122 ms. Trace:
I20250905 19:52:31.390230 2376 rpcz_store.cc:276] 0905 19:52:29.266787 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267880 (+ 1093us) service_pool.cc:225] Handling call
0905 19:52:31.303571 (+2035691us) ts_tablet_manager.cc:1145] Deleting tablet b13d282e1dc54d1faa2882c9b39d17cb
0905 19:52:31.303680 (+ 109us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.355590 (+ 51910us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.368791 (+ 13201us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.389400 (+ 20609us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2035497}
I20250905 19:52:31.391871 1170 tablet_bootstrap.cc:492] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.392063 1170 ts_tablet_manager.cc:1397] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.122s user 0.015s sys 0.019s
I20250905 19:52:31.392745 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet b13d282e1dc54d1faa2882c9b39d17cb (table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]) successfully deleted
I20250905 19:52:31.392853 1170 raft_consensus.cc:357] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.393029 1170 raft_consensus.cc:383] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.393118 1170 raft_consensus.cc:738] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.393334 1170 consensus_queue.cc:260] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.393862 1170 ts_tablet_manager.cc:1428] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:31.394220 1170 tablet_bootstrap.cc:492] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.394071 2376 tablet_replica.cc:331] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.395515 2376 raft_consensus.cc:2241] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.396596 2376 raft_consensus.cc:2270] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.396219 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:31.399930 1171 tablet_bootstrap.cc:654] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.402288 1170 tablet_bootstrap.cc:654] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.400506 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:31.405480 2376 ts_tablet_manager.cc:1905] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.407984 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:31.408253 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:31.412854 1169 tablet_bootstrap.cc:492] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.413004 1169 ts_tablet_manager.cc:1397] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.082s user 0.002s sys 0.002s
I20250905 19:52:31.413714 1169 raft_consensus.cc:357] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.413882 1169 raft_consensus.cc:383] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.413939 1169 raft_consensus.cc:738] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.414117 1169 consensus_queue.cc:260] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.414737 2379 ts_tablet_manager.cc:1918] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:31.414877 2379 log.cc:1199] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/2f581403ec924fb0948400df41ed8aab
I20250905 19:52:31.414892 1169 ts_tablet_manager.cc:1428] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:31.415355 1169 tablet_bootstrap.cc:492] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.415233 2379 ts_tablet_manager.cc:1939] T 2f581403ec924fb0948400df41ed8aab P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.417129 1169 tablet_bootstrap.cc:654] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.425684 2577 raft_consensus.cc:491] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.425848 2577 raft_consensus.cc:513] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.426460 2577 leader_election.cc:290] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.427157 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.427342 1036 raft_consensus.cc:2466] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:31.428256 849 leader_election.cc:304] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:31.430567 2577 raft_consensus.cc:2802] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.430692 2577 raft_consensus.cc:491] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.430691 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:31.431121 2577 raft_consensus.cc:3058] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.432009 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:31.432804 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]). Expected version 2 got 0
I20250905 19:52:31.433151 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:31.433566 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.434288 848 leader_election.cc:343] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.434670 1170 tablet_bootstrap.cc:492] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.434849 1170 ts_tablet_manager.cc:1397] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20250905 19:52:31.434877 2577 raft_consensus.cc:513] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.435986 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:31.436532 848 leader_election.cc:343] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.437376 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:31.437552 1036 raft_consensus.cc:3058] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.438189 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:31.439947 1170 raft_consensus.cc:357] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.440155 1170 raft_consensus.cc:383] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.440232 1170 raft_consensus.cc:738] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.440466 1036 raft_consensus.cc:2466] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:31.440479 1170 consensus_queue.cc:260] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.440953 849 leader_election.cc:304] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.441031 1170 ts_tablet_manager.cc:1428] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 19:52:31.441265 2565 raft_consensus.cc:2802] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.441390 2565 raft_consensus.cc:695] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:31.440990 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 69) took 2073 ms. Trace:
I20250905 19:52:31.441622 2565 consensus_queue.cc:237] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.444346 2577 leader_election.cc:290] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.442037 2379 rpcz_store.cc:276] 0905 19:52:29.367117 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.367210 (+ 93us) service_pool.cc:225] Handling call
0905 19:52:31.291759 (+1924549us) ts_tablet_manager.cc:1145] Deleting tablet 2f581403ec924fb0948400df41ed8aab
0905 19:52:31.291769 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.387304 (+ 95535us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.414596 (+ 27292us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.440361 (+ 25765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1923465}
I20250905 19:52:31.443786 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 2 got 0
I20250905 19:52:31.446374 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 2f581403ec924fb0948400df41ed8aab (table default.Table_929cfcb2535e452ebb48483e593faa7a [id=60675044fb6042309a7767936677de13]) successfully deleted
I20250905 19:52:31.447113 1170 tablet_bootstrap.cc:492] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.449234 1170 tablet_bootstrap.cc:654] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.451313 2568 consensus_queue.cc:1035] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.451571 2379 tablet_replica.cc:331] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:31.453344 2574 raft_consensus.cc:491] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.453497 2574 raft_consensus.cc:513] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.454066 2574 leader_election.cc:290] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.454715 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.455574 848 leader_election.cc:343] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.455596 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.455844 1044 raft_consensus.cc:2466] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:31.456370 849 leader_election.cc:304] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.457182 2574 raft_consensus.cc:2802] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.457300 2574 raft_consensus.cc:491] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.457419 2574 raft_consensus.cc:3058] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.458385 2379 raft_consensus.cc:2241] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:31.462081 2007 consensus_queue.cc:1035] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:31.466320 848 consensus_peers.cc:489] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.464522 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:31.469662 2649 tablet.cc:1722] T efceaf3c7bda44ddabd5d38130af3dd2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:31.469647 2379 raft_consensus.cc:2270] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.474541 2100 raft_consensus.cc:491] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.474720 2100 raft_consensus.cc:513] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.474900 2574 raft_consensus.cc:513] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.475364 2100 leader_election.cc:290] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.476464 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06850faa25de4b3d83f6bf98ccd370f0" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.476787 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "06850faa25de4b3d83f6bf98ccd370f0" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:31.475723 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:31.477111 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:31.477468 712 leader_election.cc:304] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:31.477648 848 leader_election.cc:343] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.478040 2100 raft_consensus.cc:2747] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.481315 1540 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_a82a3756e48a4bfc99f4aacf9b57f6d3 [id=99384aac4e624d6db630b249c3ba2543] alter complete (version 2)
I20250905 19:52:31.481686 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:31.482402 2574 leader_election.cc:290] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.483103 1539 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03] alter complete (version 2)
I20250905 19:52:31.483300 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:31.482506 2379 ts_tablet_manager.cc:1905] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.483530 1044 raft_consensus.cc:3058] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.484817 1522 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 9db56577a26e407b97fc33158c2c466c
I20250905 19:52:31.485437 1522 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 33) took 5393 ms. Trace:
I20250905 19:52:31.485543 1522 rpcz_store.cc:276] 0905 19:52:26.091632 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:28.084226 (+1992594us) service_pool.cc:225] Handling call
0905 19:52:28.084888 (+ 662us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:28.085066 (+ 178us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:28.085175 (+ 109us) catalog_manager.cc:2799] Locking tablets
0905 19:52:28.129894 (+ 44719us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:28.174069 (+ 44175us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:28.271937 (+ 97868us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:31.484751 (+3212814us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:31.485374 (+ 623us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:28.168593 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:28.168712 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:28.168717 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:28.168720 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:28.168889 (+ 169us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:28.168904 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:28.168912 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:28.169055 (+ 143us) tablet.cc:699] Row locks acquired
0905 19:52:28.169060 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:28.169110 (+ 50us) write_op.cc:270] Start()
0905 19:52:28.169151 (+ 41us) write_op.cc:276] Timestamp: P: 1757101948169078 usec, L: 0
0905 19:52:28.169157 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:28.169380 (+ 223us) log.cc:844] Serialized 725 byte log entry
0905 19:52:28.172673 (+ 3293us) op_driver.cc:464] REPLICATION: finished
0905 19:52:28.172801 (+ 128us) write_op.cc:301] APPLY: starting
0905 19:52:28.172815 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:28.172858 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:28.172894 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:28.172958 (+ 64us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:28.172989 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:28.172996 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:28.173069 (+ 73us) log.cc:844] Serialized 29 byte log entry
0905 19:52:28.173128 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.173164 (+ 36us) write_op.cc:454] Released schema lock
0905 19:52:28.173171 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:28.173253 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.173257 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":435,"apply.run_wall_time_us":434,"mutex_wait_us":6727,"num_ops":2,"prepare.queue_time_us":6780,"prepare.run_cpu_time_us":893,"prepare.run_wall_time_us":892,"raft.queue_time_us":1135,"raft.run_cpu_time_us":515,"raft.run_wall_time_us":515,"replication_time_us":3487}]]}
I20250905 19:52:31.486475 1044 raft_consensus.cc:2466] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:31.486531 1522 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 76) took 1289 ms (client timeout 1000 ms). Trace:
W20250905 19:52:31.486639 1522 rpcz_store.cc:269] 0905 19:52:30.196646 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.486260 (+1289614us) service_pool.cc:212] Skipping call since client already timed out
0905 19:52:31.486480 (+ 220us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:52:31.486922 849 leader_election.cc:304] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.487604 2577 raft_consensus.cc:2802] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.487725 2577 raft_consensus.cc:695] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:31.487927 2577 consensus_queue.cc:237] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.488600 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 9db56577a26e407b97fc33158c2c466c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.489223 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 9db56577a26e407b97fc33158c2c466c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.489871 2652 tablet_replica.cc:331] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:31.490363 2652 raft_consensus.cc:2241] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.493224 2652 raft_consensus.cc:2270] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.495257 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9db56577a26e407b97fc33158c2c466c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:31.496666 2100 raft_consensus.cc:491] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.496873 2100 raft_consensus.cc:513] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.497699 2100 leader_election.cc:290] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.498703 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:31.500093 712 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:31.500240 712 leader_election.cc:304] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:31.500739 2100 raft_consensus.cc:2747] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.501129 2652 ts_tablet_manager.cc:1905] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.507419 2574 consensus_queue.cc:1035] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.514745 2574 consensus_queue.cc:1035] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.515158 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.515362 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.515588 1022 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.516274 1016 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.516467 1020 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.516608 1025 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.516744 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.516942 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.517058 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.517303 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.518399 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.518445 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.516880 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.518579 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 813f208881ac4782874138987a710d93 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.518397 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.519629 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.519904 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.520157 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.520357 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.521195 2496 raft_consensus.cc:491] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.521350 2496 raft_consensus.cc:513] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.522168 2496 leader_election.cc:290] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.522667 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dfd0b2e972c4dd4b5b1b998cd0e0eda" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.522970 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dfd0b2e972c4dd4b5b1b998cd0e0eda" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.523334 984 leader_election.cc:343] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.523509 986 leader_election.cc:343] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.523634 986 leader_election.cc:304] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.524401 2496 raft_consensus.cc:2747] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.542820 2574 raft_consensus.cc:491] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.543083 2574 raft_consensus.cc:513] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.544435 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.544503 2574 leader_election.cc:290] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:31.545222 848 leader_election.cc:343] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.549803 2646 tablet.cc:1722] T efceaf3c7bda44ddabd5d38130af3dd2 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:31.555565 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.555826 1046 raft_consensus.cc:2466] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:31.556382 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.556430 849 leader_election.cc:304] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.557583 2565 raft_consensus.cc:2802] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.557806 2565 raft_consensus.cc:491] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.557997 2565 raft_consensus.cc:3058] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.561292 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:39172
W20250905 19:52:31.570832 711 consensus_peers.cc:489] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.572039 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.572253 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.572395 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.572542 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.571058 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.573028 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.573608 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.573819 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.573982 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.575986 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.576112 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.576301 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.576889 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.577255 1169 tablet_bootstrap.cc:492] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.577404 1169 ts_tablet_manager.cc:1397] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.162s user 0.010s sys 0.006s
I20250905 19:52:31.577769 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.578447 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.578438 1169 raft_consensus.cc:357] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.578610 1169 raft_consensus.cc:383] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.578675 1169 raft_consensus.cc:738] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.578860 1169 consensus_queue.cc:260] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.579200 2422 raft_consensus.cc:491] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.579360 1169 ts_tablet_manager.cc:1428] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:31.579385 2422 raft_consensus.cc:513] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.579478 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4c76077e4adf45a7adc736b0e69032eb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.579633 1169 tablet_bootstrap.cc:492] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.580085 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 74a7e7d9874e4d26bda3ab24a6e24adc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.580688 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.580859 2100 raft_consensus.cc:491] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.581070 2422 leader_election.cc:290] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.581036 2100 raft_consensus.cc:513] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:31.581490 848 leader_election.cc:343] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.581728 2100 leader_election.cc:290] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.581761 2496 raft_consensus.cc:491] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.581869 2496 raft_consensus.cc:513] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.582104 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4fe94d1254d04949aa25346d8e323c3b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.582999 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.583719 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:31.585429 712 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
W20250905 19:52:31.587055 848 consensus_peers.cc:489] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.588773 2609 raft_consensus.cc:491] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.588919 2609 raft_consensus.cc:513] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.591395 749 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.598009 2565 raft_consensus.cc:513] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.600623 2609 leader_election.cc:290] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.601704 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9193487b4ab74dafa76a386ddcaf674b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.602298 984 leader_election.cc:343] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.602442 2379 ts_tablet_manager.cc:1918] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.602718 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4b31443b0f840749e12435e197a204a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.602787 2379 log.cc:1199] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/00d8144bb4f14358af539b7aae326918
W20250905 19:52:31.603399 984 leader_election.cc:343] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.603781 2379 ts_tablet_manager.cc:1939] T 00d8144bb4f14358af539b7aae326918 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.611091 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.613042 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.613797 749 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.614660 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 36f966bf512d4cbe9ae406386db31533 with delete_type TABLET_DATA_DELETED (replaced by tablet 3fe2b75a3e59437c885abc18614f1da2) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.616503 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.617501 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:31.617671 711 leader_election.cc:343] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:31.617878 711 leader_election.cc:304] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:31.618333 2100 raft_consensus.cc:2747] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.617964 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 65) took 2252 ms. Trace:
I20250905 19:52:31.618681 749 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.619392 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.620069 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0edac849e8834894be2d4ff323edd13d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.618726 2379 rpcz_store.cc:276] 0905 19:52:29.366325 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.366475 (+ 150us) service_pool.cc:225] Handling call
0905 19:52:31.450301 (+2083826us) ts_tablet_manager.cc:1145] Deleting tablet 00d8144bb4f14358af539b7aae326918
0905 19:52:31.450317 (+ 16us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.537097 (+ 86780us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.602174 (+ 65077us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.617610 (+ 15436us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2080388}
I20250905 19:52:31.620976 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.621634 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 00d8144bb4f14358af539b7aae326918 (table default.Table_9ae350716fe144629f1f50ebee119d72 [id=92a9305bd65247c7912a68913af4e967]) successfully deleted
I20250905 19:52:31.622675 2496 leader_election.cc:290] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.623451 2379 tablet_replica.cc:331] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:31.624316 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.624487 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.622562 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.624895 1042 raft_consensus.cc:2466] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:31.626145 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.627116 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:39172
W20250905 19:52:31.627840 1517 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 62) took 5269 ms (client timeout 1000 ms). Trace:
I20250905 19:52:31.627926 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.632331 1169 tablet_bootstrap.cc:654] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.634342 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.635305 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.635442 2652 ts_tablet_manager.cc:1918] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:31.635247 2379 raft_consensus.cc:2241] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:31.636477 2652 log.cc:1199] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9db56577a26e407b97fc33158c2c466c
W20250905 19:52:31.645279 1517 rpcz_store.cc:269] 0905 19:52:26.358764 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:26.459015 (+100251us) service_pool.cc:225] Handling call
0905 19:52:26.466386 (+ 7371us) spinlock_profiling.cc:243] Waited 4.27 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec16f8 00007fabe1cfbc86 00007fabe1cfb9b5 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:31.627798 (+5161412us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:28.008105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:28.008260 (+ 155us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:28.008271 (+ 11us) write_op.cc:435] Acquired schema lock
0905 19:52:28.008275 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:28.008442 (+ 167us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:28.008461 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:28.008469 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:28.008600 (+ 131us) tablet.cc:699] Row locks acquired
0905 19:52:28.008605 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:28.008656 (+ 51us) write_op.cc:270] Start()
0905 19:52:28.008705 (+ 49us) write_op.cc:276] Timestamp: P: 1757101948008628 usec, L: 0
0905 19:52:28.008711 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:28.011619 (+ 2908us) log.cc:844] Serialized 901 byte log entry
0905 19:52:28.011940 (+ 321us) spinlock_profiling.cc:243] Waited 2.66 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fabe3184d62 00007fabe452d2d5 00007fabe452bdf9 00007fabe4534220 00007fabe453402d 0000000000a774dc 00007fabd773314f 00007fabd773641b 00007fabd7736298 0000000000a774dc
0905 19:52:28.016172 (+ 4232us) op_driver.cc:464] REPLICATION: finished
0905 19:52:28.016260 (+ 88us) write_op.cc:301] APPLY: starting
0905 19:52:28.016282 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:28.016334 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:28.016337 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:28.016417 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:28.016450 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:28.016456 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:28.016533 (+ 77us) log.cc:844] Serialized 29 byte log entry
0905 19:52:28.016595 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.016617 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:28.016624 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":21264,"reported_tablets":164,"spinlock_wait_cycles":9809792,"child_traces":[["op",{"apply.queue_time_us":37,"apply.run_cpu_time_us":427,"apply.run_wall_time_us":427,"num_ops":2,"prepare.queue_time_us":1552,"prepare.run_cpu_time_us":1458,"prepare.run_wall_time_us":3954,"raft.queue_time_us":1398,"raft.run_cpu_time_us":267,"raft.run_wall_time_us":268,"replication_time_us":7427,"spinlock_wait_cycles":6126976}]]}
I20250905 19:52:31.601089 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4b31443b0f840749e12435e197a204a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:31.623425 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9193487b4ab74dafa76a386ddcaf674b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.647544 986 leader_election.cc:343] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.647707 986 leader_election.cc:304] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.648072 2609 raft_consensus.cc:2747] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:31.648860 986 leader_election.cc:343] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.649009 986 leader_election.cc:304] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.649442 2609 raft_consensus.cc:2747] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:31.628438 1543 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bd9ae8fd5fdc4b539308268ba9729cd6 consensus state has no leader
I20250905 19:52:31.649845 2100 raft_consensus.cc:491] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.650056 2100 raft_consensus.cc:513] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.650885 2100 leader_election.cc:290] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.653915 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f581403ec924fb0948400df41ed8aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.653950 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 757a7d3fba8d43dda36474539b7296ba with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:16 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.630191 849 leader_election.cc:304] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.654662 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.655125 2422 raft_consensus.cc:2802] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.655265 2422 raft_consensus.cc:491] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.655407 2422 raft_consensus.cc:3058] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.655583 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.656446 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 0d4204f61929498188c18529148a8d13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.657555 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.658267 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.659003 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.659082 2422 raft_consensus.cc:513] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.659708 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 29c2b030fafd44ad9e648b2527a4ad07 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.660713 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:31.663074 2379 raft_consensus.cc:2270] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:31.661468 848 leader_election.cc:343] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.664201 712 leader_election.cc:343] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 578a6b9224cc4d8585d6c4b241762e1b
I20250905 19:52:31.659955 2422 leader_election.cc:290] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.667771 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:31.668467 848 leader_election.cc:343] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.632853 1171 tablet_bootstrap.cc:492] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.671452 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:31.671716 1171 ts_tablet_manager.cc:1397] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.292s user 0.007s sys 0.004s
I20250905 19:52:31.671895 1042 raft_consensus.cc:3058] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.672466 1171 raft_consensus.cc:357] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.672691 1171 raft_consensus.cc:383] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.672777 1171 raft_consensus.cc:738] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.673010 1171 consensus_queue.cc:260] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.673512 1171 ts_tablet_manager.cc:1428] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:31.673851 1171 tablet_bootstrap.cc:492] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.633018 2568 raft_consensus.cc:491] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.675369 2568 raft_consensus.cc:513] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.676086 2568 leader_election.cc:290] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.676173 2652 ts_tablet_manager.cc:1939] T 9db56577a26e407b97fc33158c2c466c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:31.676098 2379 ts_tablet_manager.cc:1905] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.677747 749 tablet_service.cc:1515] Processing DeleteTablet for tablet d480531ba94f4b0f9fe40e88fe91512e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.677886 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.678747 848 leader_election.cc:343] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.680094 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:31.680578 848 consensus_peers.cc:489] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.682691 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:31.684720 848 consensus_peers.cc:489] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.637832 2376 ts_tablet_manager.cc:1918] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.685057 2376 log.cc:1199] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:31.685509 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.688482 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "578a6b9224cc4d8585d6c4b241762e1b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.688985 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.689162 1046 raft_consensus.cc:2466] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:31.689970 711 leader_election.cc:304] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:31.691380 2376 ts_tablet_manager.cc:1939] T f28e403dc2f6471ebea1805fd843cfb1 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:31.690891 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.692473 2100 raft_consensus.cc:2747] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.693658 774 raft_consensus.cc:3058] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.695932 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.701884 2100 consensus_queue.cc:1035] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.707844 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.712929 2565 leader_election.cc:290] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:31.718067 1543 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46], current schema version=4): Not found: failed to reset TS proxy: No leader found: Tablet 03cf686811e041e7a001669b3d86e28a consensus state has no leader
W20250905 19:52:31.718292 848 consensus_peers.cc:489] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.720585 849 leader_election.cc:304] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.721156 2568 raft_consensus.cc:2802] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:31.721916 2568 raft_consensus.cc:491] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:31.722083 2568 raft_consensus.cc:3058] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.722960 1170 tablet_bootstrap.cc:492] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.723167 1170 ts_tablet_manager.cc:1397] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.276s user 0.065s sys 0.083s
I20250905 19:52:31.723953 1170 raft_consensus.cc:357] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.724186 1170 raft_consensus.cc:383] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.724275 1170 raft_consensus.cc:738] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.724527 1170 consensus_queue.cc:260] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.724534 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6bb79e3126074736a23b128a8e14fab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.725214 1170 ts_tablet_manager.cc:1428] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:31.725203 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.725656 2379 ts_tablet_manager.cc:1918] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.726251 2379 log.cc:1199] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:31.727097 2568 raft_consensus.cc:513] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.728262 2379 ts_tablet_manager.cc:1939] T 78076f6534814939ae8c2db9a4bd1879 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.729007 2652 tablet_replica.cc:331] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:31.729208 2652 raft_consensus.cc:2241] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.729355 2652 raft_consensus.cc:2270] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.729862 2652 ts_tablet_manager.cc:1905] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.730008 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:31.730587 848 leader_election.cc:343] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.730547 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 66) took 2363 ms. Trace:
I20250905 19:52:31.731796 2568 leader_election.cc:290] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.732652 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9db56577a26e407b97fc33158c2c466c (table default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]) successfully deleted
I20250905 19:52:31.732939 2496 raft_consensus.cc:491] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20250905 19:52:31.732972 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421b28d047124222ae9cbc603ec22cee already in progress: creating tablet
I20250905 19:52:31.733098 2496 raft_consensus.cc:513] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:31.733215 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 3609de83732d4ad8857372acfbb5c32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3609de83732d4ad8857372acfbb5c32a already in progress: creating tablet
W20250905 19:52:31.733440 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
I20250905 19:52:31.731206 2379 rpcz_store.cc:276] 0905 19:52:29.366588 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.366736 (+ 148us) service_pool.cc:225] Handling call
0905 19:52:31.622891 (+2256155us) ts_tablet_manager.cc:1145] Deleting tablet 78076f6534814939ae8c2db9a4bd1879
0905 19:52:31.622912 (+ 21us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.707672 (+ 84760us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.725132 (+ 17460us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.730356 (+ 5224us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2251841}
I20250905 19:52:31.733738 2609 raft_consensus.cc:491] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.734048 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 78076f6534814939ae8c2db9a4bd1879 (table default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]) successfully deleted
I20250905 19:52:31.735746 1171 tablet_bootstrap.cc:654] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.735846 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.734772 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 68) took 2367 ms. Trace:
I20250905 19:52:31.736353 2379 rpcz_store.cc:276] 0905 19:52:29.367026 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.367189 (+ 163us) service_pool.cc:225] Handling call
0905 19:52:31.734257 (+2367068us) ts_tablet_manager.cc:1145] Deleting tablet 421b28d047124222ae9cbc603ec22cee
0905 19:52:31.734363 (+ 106us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.734751 (+ 388us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2363686}
I20250905 19:52:31.738205 1042 raft_consensus.cc:2466] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:31.737241 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2ff206366ad4344b4df5fc6a5584f8a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.739147 984 leader_election.cc:343] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.737342 769 raft_consensus.cc:3058] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.739321 2379 tablet_replica.cc:331] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:31.739825 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421b28d047124222ae9cbc603ec22cee already in progress: creating tablet
I20250905 19:52:31.740747 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 2350bd08f977424ba94ee8fa01f66efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:31.741549 2379 raft_consensus.cc:2241] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.733877 2609 raft_consensus.cc:513] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.742398 2379 raft_consensus.cc:2270] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.743119 2609 leader_election.cc:290] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.743749 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52c99a76802b481c98f77dbd90de7cbb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:31.744097 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52c99a76802b481c98f77dbd90de7cbb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.744639 984 leader_election.cc:343] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.744774 986 leader_election.cc:343] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.744885 986 leader_election.cc:304] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.745388 2609 raft_consensus.cc:2747] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.748093 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.748449 2496 leader_election.cc:290] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.750213 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2ff206366ad4344b4df5fc6a5584f8a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.750737 986 leader_election.cc:343] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.750850 986 leader_election.cc:304] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:31.753048 1543 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 67) took 3503 ms (client timeout 1000 ms). Trace:
I20250905 19:52:31.755545 2496 raft_consensus.cc:2747] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:31.754491 1543 rpcz_store.cc:269] 0905 19:52:28.249051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:28.273752 (+ 24701us) service_pool.cc:225] Handling call
0905 19:52:31.752424 (+3478672us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":86289,"reported_tablets":217}
I20250905 19:52:31.756294 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 48) took 2491 ms. Trace:
I20250905 19:52:31.756832 2376 rpcz_store.cc:276] 0905 19:52:29.264819 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267914 (+ 3095us) service_pool.cc:225] Handling call
0905 19:52:31.393756 (+2125842us) ts_tablet_manager.cc:1145] Deleting tablet f28e403dc2f6471ebea1805fd843cfb1
0905 19:52:31.393769 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.431927 (+ 38158us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.637673 (+205746us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.756028 (+118355us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2125332}
I20250905 19:52:31.758045 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet f28e403dc2f6471ebea1805fd843cfb1 (table default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]) successfully deleted
I20250905 19:52:31.759104 2376 tablet_replica.cc:331] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.760026 2376 raft_consensus.cc:2241] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.761157 2376 raft_consensus.cc:2270] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.763000 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:31.763190 1046 raft_consensus.cc:3058] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:31.771126 1542 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 66) took 4600 ms (client timeout 1000 ms). Trace:
W20250905 19:52:31.771368 1542 rpcz_store.cc:269] 0905 19:52:27.170301 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:27.845126 (+674825us) service_pool.cc:225] Handling call
0905 19:52:31.771049 (+3925923us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":18815,"reported_tablets":207}
I20250905 19:52:31.771832 2100 raft_consensus.cc:491] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:31.772128 2100 raft_consensus.cc:513] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.772607 1542 master_service.cc:948] ReplaceTablet: received request to replace tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:31.771839 2376 ts_tablet_manager.cc:1905] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.772980 2100 leader_election.cc:290] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:31.773656 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 9db56577a26e407b97fc33158c2c466c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.776926 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:31.777120 1040 raft_consensus.cc:3058] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.777582 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.780148 1040 raft_consensus.cc:2466] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:31.781100 1046 raft_consensus.cc:2466] T 1e9b7458f96148b08962a9b82223187e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:31.782543 2379 ts_tablet_manager.cc:1905] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.789080 849 leader_election.cc:304] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.789662 2565 raft_consensus.cc:2802] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.789866 2565 raft_consensus.cc:695] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:31.790234 2565 consensus_queue.cc:237] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.793641 2496 raft_consensus.cc:491] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.793794 2496 raft_consensus.cc:513] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.794442 2496 leader_election.cc:290] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.796887 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.797473 984 leader_election.cc:343] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.797708 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.798245 986 leader_election.cc:343] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.798357 986 leader_election.cc:304] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.798938 2609 raft_consensus.cc:2747] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.805331 1038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:31.807077 1169 tablet_bootstrap.cc:492] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.807570 1169 ts_tablet_manager.cc:1397] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.228s user 0.004s sys 0.004s
I20250905 19:52:31.808665 1169 raft_consensus.cc:357] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.808957 1169 raft_consensus.cc:383] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.809736 1169 raft_consensus.cc:738] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.810078 1169 consensus_queue.cc:260] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.810750 1169 ts_tablet_manager.cc:1428] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:31.813994 2496 raft_consensus.cc:491] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.814190 2496 raft_consensus.cc:513] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.814854 2496 leader_election.cc:290] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:31.815332 848 consensus_peers.cc:489] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.815692 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.816498 1169 tablet_bootstrap.cc:492] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.816874 849 leader_election.cc:304] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.817358 2568 raft_consensus.cc:2802] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.817760 2568 raft_consensus.cc:695] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:31.817898 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:31.818066 2568 consensus_queue.cc:237] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:31.819798 984 leader_election.cc:343] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.819991 986 leader_election.cc:343] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.820115 1169 tablet_bootstrap.cc:654] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.820398 986 leader_election.cc:304] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.820905 2496 raft_consensus.cc:2747] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.821264 1542 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 49) took 4724 ms. Trace:
I20250905 19:52:31.822201 1542 rpcz_store.cc:276] 0905 19:52:27.097028 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.772493 (+4675465us) service_pool.cc:225] Handling call
0905 19:52:31.818200 (+ 45707us) spinlock_profiling.cc:243] Waited 38.5 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaed3bfb 00007fabe1d0b0e7 00007fabe1d0ae15 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:31.821208 (+ 3008us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":88525696}
I20250905 19:52:31.823066 1542 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_ad054f6da70449ecad529f69c6aa9705" } new_table_name: "default.Table_fa90bb9b631642f4b6e81bf6253dc642" modify_external_catalogs: true
I20250905 19:52:31.826305 849 leader_election.cc:304] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.826920 2007 raft_consensus.cc:2802] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:31.827033 2007 raft_consensus.cc:695] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:31.827253 2007 consensus_queue.cc:237] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.828346 2652 ts_tablet_manager.cc:1918] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:31.828567 2652 log.cc:1199] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/37894e39e6f2417fbfcd6d43f1ea2b37
I20250905 19:52:31.831092 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6bb79e3126074736a23b128a8e14fab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.842427 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.846436 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.846689 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:31.846908 1171 tablet_bootstrap.cc:492] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.847118 1171 ts_tablet_manager.cc:1397] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.173s user 0.009s sys 0.002s
I20250905 19:52:31.847920 1171 raft_consensus.cc:357] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.848244 1171 raft_consensus.cc:383] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.848352 1171 raft_consensus.cc:738] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.848683 1171 consensus_queue.cc:260] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.849277 1171 ts_tablet_manager.cc:1428] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:31.850028 1171 tablet_bootstrap.cc:492] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:31.856069 712 consensus_peers.cc:489] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:31.856349 712 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2683a36998674f739b41d2a57339954c
I20250905 19:52:31.856451 712 leader_election.cc:304] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:31.863454 2100 raft_consensus.cc:2747] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:31.865345 2413 consensus_queue.cc:1035] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.866348 2413 consensus_queue.cc:1035] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:31.868886 1546 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 63) took 4357 ms (client timeout 1000 ms). Trace:
I20250905 19:52:31.874640 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:31.877485 1171 tablet_bootstrap.cc:654] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:31.870497 1546 rpcz_store.cc:269] 0905 19:52:27.510976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:27.858121 (+347145us) service_pool.cc:225] Handling call
0905 19:52:31.868306 (+4010185us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:28.349284 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:28.349422 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:28.349431 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:28.349436 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:28.349554 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:28.349568 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:28.349573 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:28.349649 (+ 76us) tablet.cc:699] Row locks acquired
0905 19:52:28.349651 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:28.349693 (+ 42us) write_op.cc:270] Start()
0905 19:52:28.349729 (+ 36us) write_op.cc:276] Timestamp: P: 1757101948349669 usec, L: 0
0905 19:52:28.349732 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:28.349931 (+ 199us) log.cc:844] Serialized 527 byte log entry
0905 19:52:28.357268 (+ 7337us) op_driver.cc:464] REPLICATION: finished
0905 19:52:28.357411 (+ 143us) write_op.cc:301] APPLY: starting
0905 19:52:28.357429 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:28.357472 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:28.357504 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:28.357564 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:28.357600 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:28.357608 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:28.357686 (+ 78us) log.cc:844] Serialized 23 byte log entry
0905 19:52:28.357756 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.357776 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:28.357783 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:28.357854 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.357860 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":41445,"reported_tablets":172,"child_traces":[["op",{"apply.queue_time_us":87,"apply.run_cpu_time_us":436,"apply.run_wall_time_us":436,"num_ops":1,"prepare.queue_time_us":56,"prepare.run_cpu_time_us":764,"prepare.run_wall_time_us":765,"raft.queue_time_us":1248,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":269,"replication_time_us":7498}]]}
I20250905 19:52:31.886338 1546 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_2d5b20700ad74baf8138bdcadb0bef37" } modify_external_catalogs: true
W20250905 19:52:31.888497 848 consensus_peers.cc:489] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.893261 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:31.905179 2496 raft_consensus.cc:491] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.905355 2496 raft_consensus.cc:513] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.906023 2496 leader_election.cc:290] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.906589 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f34bdf651d1412690039dc6ea1095d0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.906776 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f34bdf651d1412690039dc6ea1095d0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.907203 984 leader_election.cc:343] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:31.907575 986 leader_election.cc:343] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.907677 986 leader_election.cc:304] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.907903 2609 raft_consensus.cc:2747] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.909380 1169 tablet_bootstrap.cc:492] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:31.909519 1169 ts_tablet_manager.cc:1397] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.093s user 0.003s sys 0.006s
I20250905 19:52:31.910301 1169 raft_consensus.cc:357] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.910465 1169 raft_consensus.cc:383] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.910519 1169 raft_consensus.cc:738] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.910771 1169 consensus_queue.cc:260] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.911309 1169 ts_tablet_manager.cc:1428] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:31.911712 1169 tablet_bootstrap.cc:492] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:31.913116 1546 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_2d5b20700ad74baf8138bdcadb0bef37" } modify_external_catalogs: true
I20250905 19:52:31.914355 1169 tablet_bootstrap.cc:654] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.922573 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:31.929953 2609 raft_consensus.cc:491] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:31.930119 2609 raft_consensus.cc:513] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.930737 2609 leader_election.cc:290] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.931751 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:31.931988 769 raft_consensus.cc:2466] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:31.932518 984 leader_election.cc:304] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:31.932922 2609 raft_consensus.cc:2802] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:31.933000 2609 raft_consensus.cc:491] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:31.933068 2609 raft_consensus.cc:3058] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:31.938272 2007 consensus_queue.cc:1035] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:31.939544 848 consensus_peers.cc:489] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.940305 769 raft_consensus.cc:3058] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:31.942596 2376 ts_tablet_manager.cc:1918] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:31.943457 2376 log.cc:1199] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/00ac56c449ba4b7b8dbd5e87120799d9
I20250905 19:52:31.944430 2376 ts_tablet_manager.cc:1939] T 00ac56c449ba4b7b8dbd5e87120799d9 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:31.944615 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
W20250905 19:52:31.943754 986 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2683a36998674f739b41d2a57339954c
I20250905 19:52:31.937494 2609 raft_consensus.cc:513] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.947283 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2683a36998674f739b41d2a57339954c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:31.947443 768 raft_consensus.cc:3058] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:31.949263 1171 tablet_bootstrap.cc:492] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.949708 1170 tablet_bootstrap.cc:492] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.950091 768 raft_consensus.cc:2466] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 2.
I20250905 19:52:31.951238 1170 tablet_bootstrap.cc:654] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.951996 984 leader_election.cc:304] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:31.952291 2496 raft_consensus.cc:2802] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:31.952347 1171 ts_tablet_manager.cc:1397] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.103s user 0.000s sys 0.015s
I20250905 19:52:31.953258 2379 ts_tablet_manager.cc:1918] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:31.954114 2379 log.cc:1199] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/01c20eea1b024d7f93f18f0c089efc42
I20250905 19:52:31.955118 2413 consensus_queue.cc:1035] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.955185 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 62) took 2688 ms. Trace:
I20250905 19:52:31.955492 1171 raft_consensus.cc:357] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.956173 1171 raft_consensus.cc:383] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.956583 1171 raft_consensus.cc:738] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:31.957420 986 leader_election.cc:343] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2683a36998674f739b41d2a57339954c
I20250905 19:52:31.956214 2376 rpcz_store.cc:276] 0905 19:52:29.266437 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.268108 (+ 1671us) service_pool.cc:225] Handling call
0905 19:52:31.758745 (+2490637us) ts_tablet_manager.cc:1145] Deleting tablet 00ac56c449ba4b7b8dbd5e87120799d9
0905 19:52:31.758756 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.814843 (+ 56087us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.942341 (+127498us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.954993 (+ 12652us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2490363}
I20250905 19:52:31.957649 1171 consensus_queue.cc:260] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.959322 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 00ac56c449ba4b7b8dbd5e87120799d9 (table default.Table_3fa477c60f534a63b2cd4e54fe7c5c40 [id=8ce99397ad9f43508a20590a0d175ad1]) successfully deleted
I20250905 19:52:31.958668 2379 ts_tablet_manager.cc:1939] T 01c20eea1b024d7f93f18f0c089efc42 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:31.962950 2565 consensus_queue.cc:1035] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.963197 1171 ts_tablet_manager.cc:1428] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20250905 19:52:31.964893 2609 leader_election.cc:290] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:31.965766 1171 tablet_bootstrap.cc:492] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.967099 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 50) took 2702 ms. Trace:
I20250905 19:52:31.969172 2565 consensus_queue.cc:1035] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:31.970005 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 75) took 2598 ms. Trace:
I20250905 19:52:31.968703 2376 rpcz_store.cc:276] 0905 19:52:29.265013 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.268175 (+ 3162us) service_pool.cc:225] Handling call
0905 19:52:31.959690 (+2691515us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:31.959700 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.966699 (+ 6999us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2691022}
W20250905 19:52:31.972354 1484 catalog_manager.cc:4908] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6ebab2e3698f4f60a770bde9cc019600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ebab2e3698f4f60a770bde9cc019600 already in progress: creating tablet
I20250905 19:52:31.971342 2379 rpcz_store.cc:276] 0905 19:52:29.371098 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371143 (+ 45us) service_pool.cc:225] Handling call
0905 19:52:31.739056 (+2367913us) ts_tablet_manager.cc:1145] Deleting tablet 01c20eea1b024d7f93f18f0c089efc42
0905 19:52:31.739171 (+ 115us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:31.885325 (+146154us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.952410 (+ 67085us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:31.969607 (+ 17197us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2367710}
I20250905 19:52:31.973683 2376 tablet_replica.cc:331] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:31.975538 1170 tablet_bootstrap.cc:492] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:31.975780 1170 ts_tablet_manager.cc:1397] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.026s user 0.002s sys 0.005s
I20250905 19:52:31.976292 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 01c20eea1b024d7f93f18f0c089efc42 (table default.Table_90d6fa79cea5482eab5d6b62b5d7181b [id=5f72e02cea9840619ad85f0f03e4e844]) successfully deleted
I20250905 19:52:31.976595 2379 tablet_replica.cc:331] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:31.977025 2379 raft_consensus.cc:2241] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.977358 2379 raft_consensus.cc:2270] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.976598 1170 raft_consensus.cc:357] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.978425 1170 raft_consensus.cc:383] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:31.978479 1170 raft_consensus.cc:738] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:31.978674 1170 consensus_queue.cc:260] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.979171 1170 ts_tablet_manager.cc:1428] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:31.979568 1170 tablet_bootstrap.cc:492] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:31.982404 1170 tablet_bootstrap.cc:654] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:31.982858 1171 tablet_bootstrap.cc:654] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:31.984531 848 consensus_peers.cc:489] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:31.989606 2686 raft_consensus.cc:491] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:31.989827 2686 raft_consensus.cc:513] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:31.990965 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c14dc571b586405f8fd1eaa452998978" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:31.991497 984 leader_election.cc:343] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.992372 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c14dc571b586405f8fd1eaa452998978" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:31.992847 986 leader_election.cc:343] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:31.993003 986 leader_election.cc:304] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:31.994040 2652 ts_tablet_manager.cc:1939] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:31.994547 2652 tablet_replica.cc:331] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:31.994735 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (table default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]) successfully deleted
I20250905 19:52:31.997419 2376 raft_consensus.cc:2241] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:31.997663 2376 pending_rounds.cc:70] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:31.997721 2376 pending_rounds.cc:77] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089594536103936 op_type: NO_OP noop_request { }
I20250905 19:52:31.997831 2376 raft_consensus.cc:2887] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:31.997912 2376 raft_consensus.cc:2270] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:31.998960 1171 tablet_bootstrap.cc:492] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:31.999284 1171 ts_tablet_manager.cc:1397] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.003s
I20250905 19:52:32.001144 1171 raft_consensus.cc:357] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.001598 1171 raft_consensus.cc:383] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.001796 1171 raft_consensus.cc:738] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.002336 1171 consensus_queue.cc:260] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.004366 2379 ts_tablet_manager.cc:1905] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:31.998909 2376 ts_tablet_manager.cc:1905] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.005432 1171 ts_tablet_manager.cc:1428] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:32.007090 1171 tablet_bootstrap.cc:492] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:31.994796 2652 raft_consensus.cc:2241] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:31.995631 2609 raft_consensus.cc:2747] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:31.994992 2686 leader_election.cc:290] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.012171 2652 raft_consensus.cc:2270] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.015278 2652 ts_tablet_manager.cc:1905] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.017273 1171 tablet_bootstrap.cc:654] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:32.020848 712 consensus_peers.cc:489] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.021940 1170 tablet_bootstrap.cc:492] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.022289 1170 ts_tablet_manager.cc:1397] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.001s
I20250905 19:52:32.023795 1170 raft_consensus.cc:357] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.024250 1170 raft_consensus.cc:383] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.024392 1170 raft_consensus.cc:738] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.024822 1170 consensus_queue.cc:260] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.030236 1169 tablet_bootstrap.cc:492] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.030433 1169 ts_tablet_manager.cc:1397] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.119s user 0.009s sys 0.005s
I20250905 19:52:32.031284 1169 raft_consensus.cc:357] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.031498 1169 raft_consensus.cc:383] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.031581 1169 raft_consensus.cc:738] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.031847 1169 consensus_queue.cc:260] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.032514 1169 ts_tablet_manager.cc:1428] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:32.032284 1170 ts_tablet_manager.cc:1428] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20250905 19:52:32.032857 1169 tablet_bootstrap.cc:492] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.033095 2565 raft_consensus.cc:491] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.033246 2565 raft_consensus.cc:513] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.033830 2565 leader_election.cc:290] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.034262 1169 tablet_bootstrap.cc:654] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.034785 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24bcd21f19ba465aa550102e0fccb674" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.034991 1042 raft_consensus.cc:2466] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.036453 849 leader_election.cc:304] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.036751 2568 raft_consensus.cc:491] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.036608 1170 tablet_bootstrap.cc:492] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.037479 2568 raft_consensus.cc:513] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.038059 2568 leader_election.cc:290] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.038224 2568 raft_consensus.cc:2802] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.038319 2568 raft_consensus.cc:491] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.038405 2568 raft_consensus.cc:3058] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.041219 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2d929ec75d34cddad5286d8e37fcefb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.041718 848 leader_election.cc:343] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.042865 2376 ts_tablet_manager.cc:1918] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.043387 1849 raft_consensus.cc:491] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.043547 1849 raft_consensus.cc:513] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.043751 2376 log.cc:1199] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:32.044219 1849 leader_election.cc:290] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.044973 1038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.044996 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2d929ec75d34cddad5286d8e37fcefb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.045177 1042 raft_consensus.cc:2466] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:32.045982 848 consensus_peers.cc:489] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.046144 1171 tablet_bootstrap.cc:492] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.046423 1171 ts_tablet_manager.cc:1397] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20250905 19:52:32.046854 849 leader_election.cc:304] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.048017 1171 raft_consensus.cc:357] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.048182 2577 raft_consensus.cc:2802] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.048383 2577 raft_consensus.cc:491] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.048506 2577 raft_consensus.cc:3058] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.048345 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
W20250905 19:52:32.050220 712 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 8edf4b54974c427a855eee5792d75292
I20250905 19:52:32.050364 712 leader_election.cc:304] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:32.050803 1849 raft_consensus.cc:2747] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.051121 2577 raft_consensus.cc:513] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.051576 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24bcd21f19ba465aa550102e0fccb674" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.051683 2577 leader_election.cc:290] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:32.052666 848 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.048368 1171 raft_consensus.cc:383] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.053449 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2d929ec75d34cddad5286d8e37fcefb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.053598 1171 raft_consensus.cc:738] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.054049 1171 consensus_queue.cc:260] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.053543 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:32.054620 1036 raft_consensus.cc:3058] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.057127 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f2d929ec75d34cddad5286d8e37fcefb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.058434 2565 raft_consensus.cc:491] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.058568 2565 raft_consensus.cc:513] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.059072 2565 leader_election.cc:290] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.058827 1171 ts_tablet_manager.cc:1428] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250905 19:52:32.060302 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.059665 2376 ts_tablet_manager.cc:1939] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.060537 1042 raft_consensus.cc:2466] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.061481 2652 ts_tablet_manager.cc:1918] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.061823 2652 log.cc:1199] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/578a6b9224cc4d8585d6c4b241762e1b
I20250905 19:52:32.061657 1171 tablet_bootstrap.cc:492] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.059573 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.062449 2652 ts_tablet_manager.cc:1939] T 578a6b9224cc4d8585d6c4b241762e1b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.062968 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 578a6b9224cc4d8585d6c4b241762e1b (table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]) successfully deleted
I20250905 19:52:32.063278 2007 raft_consensus.cc:491] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.063402 2007 raft_consensus.cc:513] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.063333 1170 tablet_bootstrap.cc:654] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:32.063683 848 leader_election.cc:343] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.063948 2007 leader_election.cc:290] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.064405 2652 tablet_replica.cc:331] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.064391 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6221d7d058944bdb9aa730791cf2417b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.064908 848 leader_election.cc:343] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.064954 2652 raft_consensus.cc:2241] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:32.065443 848 leader_election.cc:343] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.065691 2652 raft_consensus.cc:2270] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.066481 2652 ts_tablet_manager.cc:1905] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.066730 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6221d7d058944bdb9aa730791cf2417b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.067703 1042 raft_consensus.cc:2466] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.069415 849 leader_election.cc:304] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.071846 1036 raft_consensus.cc:2466] T f2d929ec75d34cddad5286d8e37fcefb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:32.072293 2160 raft_consensus.cc:491] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.072450 2160 raft_consensus.cc:513] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.073163 2160 leader_election.cc:290] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.074301 2007 raft_consensus.cc:2802] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.074415 2007 raft_consensus.cc:491] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.074492 2007 raft_consensus.cc:3058] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.069955 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:32.075326 1171 tablet_bootstrap.cc:654] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.077497 2007 raft_consensus.cc:513] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.078138 2007 leader_election.cc:290] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.078696 849 leader_election.cc:304] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.078779 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:32.079393 848 leader_election.cc:343] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.080160 2568 raft_consensus.cc:513] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.083482 2568 leader_election.cc:290] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.084167 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24bcd21f19ba465aa550102e0fccb674" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.082774 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.080973 2007 raft_consensus.cc:2802] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.085446 2007 raft_consensus.cc:491] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.085590 2007 raft_consensus.cc:3058] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.088141 2007 raft_consensus.cc:513] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.088809 2007 leader_election.cc:290] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.069026 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 68) took 2801 ms. Trace:
I20250905 19:52:32.089649 2376 rpcz_store.cc:276] 0905 19:52:29.267180 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.268309 (+ 1129us) service_pool.cc:225] Handling call
0905 19:52:31.973250 (+2704941us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:31.973260 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.022519 (+ 49259us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.042120 (+ 19601us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.068790 (+ 26670us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2704367}
I20250905 19:52:32.090021 2376 tablet_replica.cc:331] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.090302 2376 raft_consensus.cc:2241] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.090309 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8d38452d3a0641ffa4af5d935f8ad9f0 (table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]) successfully deleted
I20250905 19:52:32.090663 2376 raft_consensus.cc:2270] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.091192 2376 ts_tablet_manager.cc:1905] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.091436 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:32.092324 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
W20250905 19:52:32.092655 712 consensus_peers.cc:489] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.092864 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:32.073829 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8503f6918ffb44efa36214633c93e31d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.082219 1038 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.093323 1038 raft_consensus.cc:3058] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.093729 711 leader_election.cc:304] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:32.094019 2160 raft_consensus.cc:2747] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.094379 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:32.084534 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "24bcd21f19ba465aa550102e0fccb674" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.095871 2379 ts_tablet_manager.cc:1918] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.096103 2379 log.cc:1199] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/54628965b82e470d811eff95ffc56c5f
I20250905 19:52:32.089768 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6221d7d058944bdb9aa730791cf2417b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.096312 1040 raft_consensus.cc:3058] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.096766 1038 raft_consensus.cc:2466] T 4d6f2e44855d4312b1c0367145e3cee8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:32.096922 2379 ts_tablet_manager.cc:1939] T 54628965b82e470d811eff95ffc56c5f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.099515 1040 raft_consensus.cc:2466] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:32.097685 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 79) took 2725 ms. Trace:
W20250905 19:52:32.084934 848 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.095108 849 leader_election.cc:304] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.101263 2568 raft_consensus.cc:2802] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.101012 2379 rpcz_store.cc:276] 0905 19:52:29.372211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.372307 (+ 96us) service_pool.cc:225] Handling call
0905 19:52:31.976166 (+2603859us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:31.976179 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.034485 (+ 58306us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.095540 (+ 61055us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.097627 (+ 2087us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2601989}
I20250905 19:52:32.101447 1171 tablet_bootstrap.cc:492] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.101553 1171 ts_tablet_manager.cc:1397] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.040s user 0.007s sys 0.000s
I20250905 19:52:32.101773 2379 tablet_replica.cc:331] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.101473 2568 raft_consensus.cc:695] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.101966 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 54628965b82e470d811eff95ffc56c5f (table default.Table_c815e30da2544766834b52eaefab63c4 [id=0b36fe8f4d004271aa9bbe78a44dc583]) successfully deleted
W20250905 19:52:32.102324 849 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: Tablet not RUNNING: STOPPED
I20250905 19:52:32.102459 849 leader_election.cc:304] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.102351 2568 consensus_queue.cc:237] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.103017 2422 raft_consensus.cc:2747] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.103353 849 leader_election.cc:304] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.103602 2422 raft_consensus.cc:2802] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.103721 2422 raft_consensus.cc:695] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.104008 2422 consensus_queue.cc:237] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.104961 849 leader_election.cc:304] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.105237 2565 raft_consensus.cc:2802] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.105638 2565 raft_consensus.cc:695] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.106141 2565 consensus_queue.cc:237] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.102392 1171 raft_consensus.cc:357] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.107323 1171 raft_consensus.cc:383] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.107483 1171 raft_consensus.cc:738] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.107739 1171 consensus_queue.cc:260] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.091215 2652 ts_tablet_manager.cc:1918] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.108403 2652 log.cc:1199] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/24bcd21f19ba465aa550102e0fccb674
I20250905 19:52:32.108597 1171 ts_tablet_manager.cc:1428] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 19:52:32.102171 2379 raft_consensus.cc:2241] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.110216 1170 tablet_bootstrap.cc:492] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.110356 1170 ts_tablet_manager.cc:1397] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.075s user 0.004s sys 0.001s
I20250905 19:52:32.111135 1170 raft_consensus.cc:357] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.111317 1170 raft_consensus.cc:383] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.111375 1170 raft_consensus.cc:738] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.111580 1170 consensus_queue.cc:260] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.112156 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6221d7d058944bdb9aa730791cf2417b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.112846 1171 tablet_bootstrap.cc:492] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.109027 2652 ts_tablet_manager.cc:1939] T 24bcd21f19ba465aa550102e0fccb674 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.113175 1170 ts_tablet_manager.cc:1428] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:32.113823 2379 raft_consensus.cc:2270] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.114468 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 24bcd21f19ba465aa550102e0fccb674 (table default.Table_65d1d80076a24f24a1c5284f9e90ec10 [id=c7b558dcabda400d9ba6e1850e0a61eb]) successfully deleted
I20250905 19:52:32.115231 2652 tablet_replica.cc:331] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.115842 2652 raft_consensus.cc:2241] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.116345 1171 tablet_bootstrap.cc:654] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.116590 2652 raft_consensus.cc:2270] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.117478 2652 ts_tablet_manager.cc:1905] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.116576 1170 tablet_bootstrap.cc:492] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:32.113710 848 leader_election.cc:343] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.121524 2686 raft_consensus.cc:491] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:32.121690 2686 raft_consensus.cc:513] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.122421 2686 leader_election.cc:290] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.122702 2379 ts_tablet_manager.cc:1905] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.123786 986 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:32.123117 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.124073 768 raft_consensus.cc:2466] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:32.124612 984 leader_election.cc:304] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:32.125195 2686 raft_consensus.cc:2802] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:32.125311 2686 raft_consensus.cc:491] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:32.125417 2686 raft_consensus.cc:3058] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:32.124328 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:32.128834 2686 raft_consensus.cc:513] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.130040 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e5ca7bd3e84916a11e01afbf48171d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.130231 768 raft_consensus.cc:3058] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 2
I20250905 19:52:32.133057 768 raft_consensus.cc:2466] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 2.
I20250905 19:52:32.133584 984 leader_election.cc:304] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:32.133807 2609 raft_consensus.cc:2802] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:32.133914 2609 raft_consensus.cc:695] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:32.134128 2609 consensus_queue.cc:237] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:32.135035 986 leader_election.cc:343] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:32.135192 2686 leader_election.cc:290] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.138269 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:32.148020 1170 tablet_bootstrap.cc:654] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.149931 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:32.154623 2496 raft_consensus.cc:695] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:32.154932 2496 consensus_queue.cc:237] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.155349 2422 consensus_queue.cc:1035] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:32.171149 1849 consensus_queue.cc:1035] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:32.179483 2652 ts_tablet_manager.cc:1918] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.183832 2376 ts_tablet_manager.cc:1918] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.184011 2376 log.cc:1199] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:32.184409 2376 ts_tablet_manager.cc:1939] T 9630f1feb6d74472b5f6f4b54151311c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.186419 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 67) took 2919 ms. Trace:
I20250905 19:52:32.186544 2376 rpcz_store.cc:276] 0905 19:52:29.267035 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267613 (+ 578us) service_pool.cc:225] Handling call
0905 19:52:32.089973 (+2822360us) ts_tablet_manager.cc:1145] Deleting tablet 9630f1feb6d74472b5f6f4b54151311c
0905 19:52:32.089983 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.178592 (+ 88609us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.181916 (+ 3324us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.186390 (+ 4474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2820846}
I20250905 19:52:32.186950 2376 tablet_replica.cc:331] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.187160 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9630f1feb6d74472b5f6f4b54151311c (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]) successfully deleted
I20250905 19:52:32.188648 2376 raft_consensus.cc:2241] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.193818 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:32.194288 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:32.195006 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:32.195130 2376 raft_consensus.cc:2270] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.197556 2376 ts_tablet_manager.cc:1905] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.198915 711 consensus_peers.cc:489] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.201004 2652 log.cc:1199] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a1b4989f17454aa58c05727b797c2009
I20250905 19:52:32.204876 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:32.211198 1171 tablet_bootstrap.cc:492] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.211360 1171 ts_tablet_manager.cc:1397] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.103s user 0.004s sys 0.000s
I20250905 19:52:32.211957 1170 tablet_bootstrap.cc:492] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.212123 1170 ts_tablet_manager.cc:1397] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.097s user 0.004s sys 0.002s
I20250905 19:52:32.212234 1171 raft_consensus.cc:357] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.212437 1171 raft_consensus.cc:383] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.212514 1171 raft_consensus.cc:738] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.213400 1170 raft_consensus.cc:357] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.213575 1170 raft_consensus.cc:383] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.213642 1170 raft_consensus.cc:738] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.214038 1170 consensus_queue.cc:260] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.216699 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:32.217784 1170 ts_tablet_manager.cc:1428] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:32.218835 2652 ts_tablet_manager.cc:1939] T a1b4989f17454aa58c05727b797c2009 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.220003 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a1b4989f17454aa58c05727b797c2009 (table default.Table_ee3eaf9eb84d435187c73a394308861b [id=8225e2fcccbf436db5bfa9e05ba88f83]) successfully deleted
I20250905 19:52:32.221004 2376 ts_tablet_manager.cc:1918] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.221125 2376 log.cc:1199] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:32.221514 2376 ts_tablet_manager.cc:1939] T ac5c88cf8d15433a940e937f0a225582 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.221813 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 53) took 2956 ms. Trace:
I20250905 19:52:32.221906 2376 rpcz_store.cc:276] 0905 19:52:29.265311 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.267709 (+ 2398us) service_pool.cc:225] Handling call
0905 19:52:32.186887 (+2919178us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:32.186900 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.202238 (+ 15338us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.220882 (+ 18644us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.221799 (+ 917us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":1289,"tablet-delete.queue_time_us":2919016}
I20250905 19:52:32.222352 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ac5c88cf8d15433a940e937f0a225582 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:32.222813 1170 tablet_bootstrap.cc:492] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.224009 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
W20250905 19:52:32.226248 712 consensus_peers.cc:489] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.226578 2652 tablet_replica.cc:331] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.226845 2652 raft_consensus.cc:2241] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.227052 2652 raft_consensus.cc:2270] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.227661 2652 ts_tablet_manager.cc:1905] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.234711 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:32.237936 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:32.241279 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:32.243479 2379 ts_tablet_manager.cc:1918] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.245337 2379 log.cc:1199] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/5b0d4be788f841c2860f5dd65275b623
I20250905 19:52:32.245136 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:32.247179 769 raft_consensus.cc:3058] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.255162 1171 consensus_queue.cc:260] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.256613 1169 tablet_bootstrap.cc:492] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.256757 1169 ts_tablet_manager.cc:1397] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.224s user 0.008s sys 0.018s
I20250905 19:52:32.257021 2496 raft_consensus.cc:491] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.257174 1171 ts_tablet_manager.cc:1428] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.046s user 0.003s sys 0.000s
I20250905 19:52:32.257202 2496 raft_consensus.cc:513] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.257520 1171 tablet_bootstrap.cc:492] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.249444 1170 tablet_bootstrap.cc:654] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.258503 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.249191 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:32.259436 1171 tablet_bootstrap.cc:654] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:32.259657 984 leader_election.cc:343] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.260064 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.259311 2379 ts_tablet_manager.cc:1939] T 5b0d4be788f841c2860f5dd65275b623 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:32.260485 986 leader_election.cc:343] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.260592 986 leader_election.cc:304] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.262141 2685 raft_consensus.cc:2747] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.262388 2496 leader_election.cc:290] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.263262 1169 raft_consensus.cc:357] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.263545 1169 raft_consensus.cc:383] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.265578 2713 tablet.cc:1722] T efceaf3c7bda44ddabd5d38130af3dd2 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:32.265908 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 77) took 2894 ms. Trace:
I20250905 19:52:32.266810 2379 rpcz_store.cc:276] 0905 19:52:29.371302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371674 (+ 372us) service_pool.cc:225] Handling call
0905 19:52:32.101651 (+2729977us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:32.101694 (+ 43us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.191272 (+ 89578us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.242448 (+ 51176us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.265653 (+ 23205us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2728162}
I20250905 19:52:32.263743 1169 raft_consensus.cc:738] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.269693 1169 consensus_queue.cc:260] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.269824 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 5b0d4be788f841c2860f5dd65275b623 (table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]) successfully deleted
I20250905 19:52:32.270502 1169 ts_tablet_manager.cc:1428] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250905 19:52:32.270793 1169 tablet_bootstrap.cc:492] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.272230 1169 tablet_bootstrap.cc:654] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.272022 2379 tablet_replica.cc:331] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.274127 2379 raft_consensus.cc:2241] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.273814 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:32.277233 2379 raft_consensus.cc:2270] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.281422 2652 ts_tablet_manager.cc:1918] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.281634 2652 log.cc:1199] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:32.282011 2652 ts_tablet_manager.cc:1939] T 52c99a76802b481c98f77dbd90de7cbb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.282356 2376 tablet_replica.cc:331] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.284024 2376 raft_consensus.cc:2241] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.284202 2376 raft_consensus.cc:2270] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.285032 2376 ts_tablet_manager.cc:1905] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.286100 848 consensus_peers.cc:489] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.286819 2379 ts_tablet_manager.cc:1905] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.287720 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:32.291718 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:32.294799 2376 ts_tablet_manager.cc:1918] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.294925 2376 log.cc:1199] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:32.295292 2376 ts_tablet_manager.cc:1939] T 03a12cf719f34096bb82f5846bbe3b2d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.295603 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 63) took 3029 ms. Trace:
I20250905 19:52:32.296541 1170 tablet_bootstrap.cc:492] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.296692 1170 ts_tablet_manager.cc:1397] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.002s
I20250905 19:52:32.295686 2376 rpcz_store.cc:276] 0905 19:52:29.266544 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.268982 (+ 2438us) service_pool.cc:225] Handling call
0905 19:52:32.273688 (+3004706us) ts_tablet_manager.cc:1145] Deleting tablet 03a12cf719f34096bb82f5846bbe3b2d
0905 19:52:32.282292 (+ 8604us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.291634 (+ 9342us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.294700 (+ 3066us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.295587 (+ 887us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3004478}
I20250905 19:52:32.302986 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 03a12cf719f34096bb82f5846bbe3b2d (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]) successfully deleted
I20250905 19:52:32.305699 2376 tablet_replica.cc:331] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.305968 2376 raft_consensus.cc:2241] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.306396 2376 raft_consensus.cc:2270] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.307026 2376 ts_tablet_manager.cc:1905] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.309792 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 52c99a76802b481c98f77dbd90de7cbb (table default.Table_29345948e72a432f9d41b63ca0c45110 [id=7ae4e86b2cb6498f90c5a0f9c662a82f]) successfully deleted
I20250905 19:52:32.310000 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:32.310165 2652 tablet_replica.cc:331] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.310416 2652 raft_consensus.cc:2241] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.310632 2652 raft_consensus.cc:2270] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.311193 2652 ts_tablet_manager.cc:1905] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.312031 1169 tablet_bootstrap.cc:492] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.318902 1169 ts_tablet_manager.cc:1397] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250905 19:52:32.319789 1169 raft_consensus.cc:357] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.320020 1169 raft_consensus.cc:383] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.320103 1169 raft_consensus.cc:738] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.320344 1169 consensus_queue.cc:260] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.320904 1169 ts_tablet_manager.cc:1428] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:32.320827 1170 raft_consensus.cc:357] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.321014 1170 raft_consensus.cc:383] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.321071 1170 raft_consensus.cc:738] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.321254 1170 consensus_queue.cc:260] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.321727 1170 ts_tablet_manager.cc:1428] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.025s user 0.002s sys 0.000s
I20250905 19:52:32.321835 1169 tablet_bootstrap.cc:492] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.322393 1849 raft_consensus.cc:491] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.322507 1849 raft_consensus.cc:513] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.323105 1849 leader_election.cc:290] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.323796 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.324751 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e58bbb6575fd4437b242032a281914ea" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.325623 1169 tablet_bootstrap.cc:654] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.325594 711 leader_election.cc:304] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:32.326021 1849 raft_consensus.cc:2747] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.327402 1171 tablet_bootstrap.cc:492] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.327576 1171 ts_tablet_manager.cc:1397] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250905 19:52:32.328363 1171 raft_consensus.cc:357] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.334764 2413 consensus_queue.cc:1035] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:32.328540 1171 raft_consensus.cc:383] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.335651 1171 raft_consensus.cc:738] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.335942 1171 consensus_queue.cc:260] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.339056 1171 ts_tablet_manager.cc:1428] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.001s sys 0.000s
I20250905 19:52:32.336337 2376 ts_tablet_manager.cc:1918] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.339284 2376 log.cc:1199] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/5b0d4be788f841c2860f5dd65275b623
I20250905 19:52:32.339735 2376 ts_tablet_manager.cc:1939] T 5b0d4be788f841c2860f5dd65275b623 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.342406 1171 tablet_bootstrap.cc:492] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.344554 1171 tablet_bootstrap.cc:654] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.345417 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 66) took 3078 ms. Trace:
I20250905 19:52:32.345496 2376 rpcz_store.cc:276] 0905 19:52:29.266920 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.269247 (+ 2327us) service_pool.cc:225] Handling call
0905 19:52:32.305645 (+3036398us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:32.305657 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.319290 (+ 13633us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.336235 (+ 16945us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.345398 (+ 9163us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3036211}
I20250905 19:52:32.345989 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 5b0d4be788f841c2860f5dd65275b623 (table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]) successfully deleted
I20250905 19:52:32.346725 2376 tablet_replica.cc:331] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.346930 2376 raft_consensus.cc:2241] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.347064 2376 raft_consensus.cc:2270] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.347528 2376 ts_tablet_manager.cc:1905] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.354278 1171 tablet_bootstrap.cc:492] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.354494 1171 ts_tablet_manager.cc:1397] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20250905 19:52:32.355890 1171 raft_consensus.cc:357] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.356174 1171 raft_consensus.cc:383] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.356272 1171 raft_consensus.cc:738] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.356544 1171 consensus_queue.cc:260] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.359092 1170 tablet_bootstrap.cc:492] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.360852 1170 tablet_bootstrap.cc:654] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:32.368849 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46], current schema version=4) failed: Not found: failed to reset TS proxy: No leader found: Tablet 03cf686811e041e7a001669b3d86e28a consensus state has no leader
I20250905 19:52:32.371417 2379 ts_tablet_manager.cc:1918] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.373191 1169 tablet_bootstrap.cc:492] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.373021 2379 log.cc:1199] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/05ae83eea5f74e9187713e8b82c9a23b
W20250905 19:52:32.374225 848 consensus_peers.cc:489] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.377998 2652 ts_tablet_manager.cc:1918] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.380908 2379 ts_tablet_manager.cc:1939] T 05ae83eea5f74e9187713e8b82c9a23b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.385075 2376 ts_tablet_manager.cc:1918] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.387683 2652 log.cc:1199] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:32.392808 2652 ts_tablet_manager.cc:1939] T fd793bce71e344f18bb177a53ab1bfc1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.374533 1169 ts_tablet_manager.cc:1397] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.053s user 0.004s sys 0.000s
I20250905 19:52:32.392907 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 73) took 3021 ms. Trace:
I20250905 19:52:32.395998 1169 raft_consensus.cc:357] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.396238 1169 raft_consensus.cc:383] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.396337 1169 raft_consensus.cc:738] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.394326 1171 ts_tablet_manager.cc:1428] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.040s user 0.002s sys 0.000s
I20250905 19:52:32.396636 1169 consensus_queue.cc:260] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.397670 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet fd793bce71e344f18bb177a53ab1bfc1 (table default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 [id=1e2786d48c8d49cfa9cbca17c164b7eb]) successfully deleted
I20250905 19:52:32.395601 2379 rpcz_store.cc:276] 0905 19:52:29.370926 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371746 (+ 820us) service_pool.cc:225] Handling call
0905 19:52:32.271489 (+2899743us) ts_tablet_manager.cc:1145] Deleting tablet 05ae83eea5f74e9187713e8b82c9a23b
0905 19:52:32.271738 (+ 249us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.316782 (+ 45044us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.370577 (+ 53795us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.392567 (+ 21990us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2897687}
I20250905 19:52:32.398566 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 05ae83eea5f74e9187713e8b82c9a23b (table default.Table_ab2b806012994ad38e6f7a3171f00777 [id=29f567c56184458c96d24eb78f402137]) successfully deleted
I20250905 19:52:32.399274 2652 tablet_replica.cc:331] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.399494 2652 raft_consensus.cc:2241] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.396790 1171 tablet_bootstrap.cc:492] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.396953 2376 log.cc:1199] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:32.400316 2376 ts_tablet_manager.cc:1939] T 2350bd08f977424ba94ee8fa01f66efa P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.400092 2379 tablet_replica.cc:331] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.401124 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 31) took 3237 ms. Trace:
I20250905 19:52:32.403163 2652 raft_consensus.cc:2270] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.403213 2379 raft_consensus.cc:2241] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.401219 2376 rpcz_store.cc:276] 0905 19:52:29.163506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.244511 (+ 81005us) service_pool.cc:225] Handling call
0905 19:52:32.346678 (+3102167us) ts_tablet_manager.cc:1145] Deleting tablet 2350bd08f977424ba94ee8fa01f66efa
0905 19:52:32.346687 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.350867 (+ 4180us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.384962 (+ 34095us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.401102 (+ 16140us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3076718}
I20250905 19:52:32.398334 1169 ts_tablet_manager.cc:1428] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:32.404592 1169 tablet_bootstrap.cc:492] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.404889 2376 tablet_replica.cc:331] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.405073 2376 raft_consensus.cc:2241] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.405124 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 2350bd08f977424ba94ee8fa01f66efa (table default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]) successfully deleted
I20250905 19:52:32.405227 2376 raft_consensus.cc:2270] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.405337 2379 raft_consensus.cc:2270] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.406865 2376 ts_tablet_manager.cc:1905] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.411059 1169 tablet_bootstrap.cc:654] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.412405 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:32.402787 1171 tablet_bootstrap.cc:654] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.414680 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:32.415797 2568 raft_consensus.cc:491] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.415923 2568 raft_consensus.cc:513] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.416525 2568 leader_election.cc:290] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.416651 2379 ts_tablet_manager.cc:1905] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.417542 849 leader_election.cc:343] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:32.417580 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:32.417843 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52c99a76802b481c98f77dbd90de7cbb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.418326 848 leader_election.cc:343] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.418442 848 leader_election.cc:304] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.418977 2577 raft_consensus.cc:2747] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.420259 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:32.422202 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:32.423261 1170 tablet_bootstrap.cc:492] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.423442 1170 ts_tablet_manager.cc:1397] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.064s user 0.007s sys 0.003s
I20250905 19:52:32.423871 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:32.424289 1170 raft_consensus.cc:357] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.424467 1170 raft_consensus.cc:383] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.424575 1170 raft_consensus.cc:738] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.424786 1170 consensus_queue.cc:260] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.411301 2652 ts_tablet_manager.cc:1905] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.425267 1170 ts_tablet_manager.cc:1428] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:32.425537 1170 tablet_bootstrap.cc:492] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.427129 1170 tablet_bootstrap.cc:654] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.439271 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:32.443074 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:32.445176 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:32.446280 2376 ts_tablet_manager.cc:1918] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.446640 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:32.447331 1170 tablet_bootstrap.cc:492] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.447521 1170 ts_tablet_manager.cc:1397] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.001s
I20250905 19:52:32.447449 1171 tablet_bootstrap.cc:492] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.447870 1171 ts_tablet_manager.cc:1397] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.051s user 0.005s sys 0.000s
I20250905 19:52:32.448464 1170 raft_consensus.cc:357] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.448670 1170 raft_consensus.cc:383] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.448848 1170 raft_consensus.cc:738] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.449136 1170 consensus_queue.cc:260] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.449692 1170 ts_tablet_manager.cc:1428] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:32.449780 2376 log.cc:1199] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:32.450176 1170 tablet_bootstrap.cc:492] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.451864 1170 tablet_bootstrap.cc:654] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.452622 2376 ts_tablet_manager.cc:1939] T 78076f6534814939ae8c2db9a4bd1879 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.454360 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:32.454766 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 47) took 3190 ms. Trace:
I20250905 19:52:32.455592 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:32.456156 2376 rpcz_store.cc:276] 0905 19:52:29.264703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.270656 (+ 5953us) service_pool.cc:225] Handling call
0905 19:52:32.404842 (+3134186us) ts_tablet_manager.cc:1145] Deleting tablet 78076f6534814939ae8c2db9a4bd1879
0905 19:52:32.404852 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.437092 (+ 32240us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.446162 (+ 9070us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.454745 (+ 8583us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3134008}
I20250905 19:52:32.456777 2376 tablet_replica.cc:331] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.457044 2376 raft_consensus.cc:2241] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.456833 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:32.457662 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 78076f6534814939ae8c2db9a4bd1879 (table default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]) successfully deleted
I20250905 19:52:32.457253 2376 raft_consensus.cc:2270] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.458313 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:32.459203 2376 ts_tablet_manager.cc:1905] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.459257 1169 tablet_bootstrap.cc:492] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.459389 1169 ts_tablet_manager.cc:1397] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.000s
I20250905 19:52:32.460093 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:32.460808 1169 raft_consensus.cc:357] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.460999 1169 raft_consensus.cc:383] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.461077 1169 raft_consensus.cc:738] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.461333 1169 consensus_queue.cc:260] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.461898 1169 ts_tablet_manager.cc:1428] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:32.461053 1517 catalog_manager.cc:5582] T 3587780819704008b7f4ee6abc714db6 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:32.462059 769 raft_consensus.cc:3058] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.465487 1169 tablet_bootstrap.cc:492] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.464097 1517 catalog_manager.cc:5582] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:32.457118 1171 raft_consensus.cc:357] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.469489 1171 raft_consensus.cc:383] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.469592 1171 raft_consensus.cc:738] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.469846 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:32.470578 1171 consensus_queue.cc:260] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.471501 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:32.472730 2652 ts_tablet_manager.cc:1918] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.472994 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:32.473891 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:32.474946 2652 log.cc:1199] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d296428b70f9411190b6c408d81639f2
I20250905 19:52:32.474967 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:32.475337 2652 ts_tablet_manager.cc:1939] T d296428b70f9411190b6c408d81639f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.477248 1171 ts_tablet_manager.cc:1428] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250905 19:52:32.479344 1169 tablet_bootstrap.cc:654] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.480288 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d296428b70f9411190b6c408d81639f2 (table default.Table_63b0eb8b21074d33a20cc13f66b38ad9 [id=80e5ee2179c744f7beccf76fc5ceb396]) successfully deleted
I20250905 19:52:32.480443 1171 tablet_bootstrap.cc:492] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.483305 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:32.485260 1171 tablet_bootstrap.cc:654] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.486050 1170 tablet_bootstrap.cc:492] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.486976 1170 ts_tablet_manager.cc:1397] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.037s user 0.004s sys 0.000s
I20250905 19:52:32.487020 1517 catalog_manager.cc:5582] T fb72fe6049894527917e1481bcb7b03a P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:32.490233 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:32.491890 1170 raft_consensus.cc:357] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.492971 1170 raft_consensus.cc:383] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.493428 1170 raft_consensus.cc:738] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.494469 1170 consensus_queue.cc:260] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.493255 1517 catalog_manager.cc:5582] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
W20250905 19:52:32.496521 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6ebab2e3698f4f60a770bde9cc019600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ebab2e3698f4f60a770bde9cc019600 already in progress: creating tablet
I20250905 19:52:32.496165 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:32.496860 2652 tablet_replica.cc:331] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.497095 2652 raft_consensus.cc:2241] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.497273 2652 raft_consensus.cc:2270] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.497869 2652 ts_tablet_manager.cc:1905] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.498106 1170 ts_tablet_manager.cc:1428] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.011s user 0.000s sys 0.003s
I20250905 19:52:32.499038 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:32.499763 1170 tablet_bootstrap.cc:492] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.500406 2379 ts_tablet_manager.cc:1918] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.501737 2376 ts_tablet_manager.cc:1918] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.501909 2376 log.cc:1199] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:32.501842 2379 log.cc:1199] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9e9994863c374354838aa0f817d0057e
I20250905 19:52:32.504841 2376 ts_tablet_manager.cc:1939] T 06850faa25de4b3d83f6bf98ccd370f0 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.505198 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 45) took 3240 ms. Trace:
I20250905 19:52:32.505411 2376 rpcz_store.cc:276] 0905 19:52:29.264443 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.270902 (+ 6459us) service_pool.cc:225] Handling call
0905 19:52:32.456722 (+3185820us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:32.456734 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.495745 (+ 39011us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.501640 (+ 5895us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.505181 (+ 3541us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3185610}
I20250905 19:52:32.506206 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 06850faa25de4b3d83f6bf98ccd370f0 (table default.Table_4ec0491dbe8841049cceaeee7c721211 [id=23b4c9dd0ca7459fa0d0a8b5f7033fd9]) successfully deleted
I20250905 19:52:32.506275 2376 tablet_replica.cc:331] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.507300 2379 ts_tablet_manager.cc:1939] T 9e9994863c374354838aa0f817d0057e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.504227 1517 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:32.523559 1169 tablet_bootstrap.cc:492] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.523794 1169 ts_tablet_manager.cc:1397] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.002s
I20250905 19:52:32.524231 2376 raft_consensus.cc:2241] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.524786 2376 raft_consensus.cc:2270] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.524745 1169 raft_consensus.cc:357] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.525018 1169 raft_consensus.cc:383] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.525127 1169 raft_consensus.cc:738] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.525444 1169 consensus_queue.cc:260] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.526099 1169 ts_tablet_manager.cc:1428] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:32.526393 1169 tablet_bootstrap.cc:492] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.527621 2568 raft_consensus.cc:491] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.527765 2568 raft_consensus.cc:513] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.528440 2568 leader_election.cc:290] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.529137 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dfd0b2e972c4dd4b5b1b998cd0e0eda" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.529230 2376 ts_tablet_manager.cc:1905] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.529320 1037 raft_consensus.cc:2466] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.528687 1170 tablet_bootstrap.cc:654] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.530004 849 leader_election.cc:304] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.530339 2577 raft_consensus.cc:2802] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.530401 2577 raft_consensus.cc:491] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.530454 2577 raft_consensus.cc:3058] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.532704 2577 raft_consensus.cc:513] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.533272 2577 leader_election.cc:290] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.534457 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dfd0b2e972c4dd4b5b1b998cd0e0eda" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.534643 1037 raft_consensus.cc:3058] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.536294 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:32.537767 1037 raft_consensus.cc:2466] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:32.539829 849 consensus_peers.cc:489] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37894e39e6f2417fbfcd6d43f1ea2b37. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.540053 849 leader_election.cc:304] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.540346 2568 raft_consensus.cc:2802] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.540494 2568 raft_consensus.cc:695] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.540798 2568 consensus_queue.cc:237] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.546751 2376 ts_tablet_manager.cc:1918] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.548274 2684 raft_consensus.cc:491] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:32.548442 2684 raft_consensus.cc:513] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.549165 2684 leader_election.cc:290] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.549912 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dfd0b2e972c4dd4b5b1b998cd0e0eda" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.550233 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9dfd0b2e972c4dd4b5b1b998cd0e0eda" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.550684 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.550765 986 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 8edf4b54974c427a855eee5792d75292
I20250905 19:52:32.550832 770 raft_consensus.cc:2466] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:32.551437 984 leader_election.cc:304] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:32.552066 2685 raft_consensus.cc:2802] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:32.552156 2685 raft_consensus.cc:491] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:32.552244 2685 raft_consensus.cc:3058] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:32.553449 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 78) took 3181 ms. Trace:
I20250905 19:52:32.553552 2379 rpcz_store.cc:276] 0905 19:52:29.372081 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.372120 (+ 39us) service_pool.cc:225] Handling call
0905 19:52:32.399741 (+3027621us) ts_tablet_manager.cc:1145] Deleting tablet 9e9994863c374354838aa0f817d0057e
0905 19:52:32.399754 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.461117 (+ 61363us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.499037 (+ 37920us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.553427 (+ 54390us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":384,"tablet-delete.queue_time_us":3025432}
I20250905 19:52:32.555181 2685 raft_consensus.cc:513] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.555886 2685 leader_election.cc:290] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.556298 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9e9994863c374354838aa0f817d0057e (table default.Table_56e29cf512764408b243635ae01a6ab0 [id=aad015c74608490985ee75fbd380f0e3]) successfully deleted
W20250905 19:52:32.556792 986 leader_election.cc:343] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 8edf4b54974c427a855eee5792d75292
I20250905 19:52:32.555806 2376 log.cc:1199] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/42f3c1efb94e446fb0128b6b412821e2
W20250905 19:52:32.557008 848 leader_election.cc:343] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:32.557533 848 leader_election.cc:343] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.557593 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8edf4b54974c427a855eee5792d75292" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.557778 770 raft_consensus.cc:3058] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 2
I20250905 19:52:32.557914 2379 tablet_replica.cc:331] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.558121 2379 raft_consensus.cc:2241] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.558266 2379 raft_consensus.cc:2270] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.559031 2376 ts_tablet_manager.cc:1939] T 42f3c1efb94e446fb0128b6b412821e2 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.561221 770 raft_consensus.cc:2466] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 2.
I20250905 19:52:32.561287 2379 ts_tablet_manager.cc:1905] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.561661 984 leader_election.cc:304] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:32.562076 2684 raft_consensus.cc:2802] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:32.562193 2684 raft_consensus.cc:695] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:32.562430 2684 consensus_queue.cc:237] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.567498 2379 ts_tablet_manager.cc:1918] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.567639 2379 log.cc:1199] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ab4f69c45295422f85b310aa55220e9b
I20250905 19:52:32.568935 1170 tablet_bootstrap.cc:492] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.569065 1170 ts_tablet_manager.cc:1397] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.070s user 0.006s sys 0.000s
I20250905 19:52:32.569802 1170 raft_consensus.cc:357] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.569968 1170 raft_consensus.cc:383] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.570016 1170 raft_consensus.cc:738] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.570191 1170 consensus_queue.cc:260] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.570677 1170 ts_tablet_manager.cc:1428] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:32.570981 1170 tablet_bootstrap.cc:492] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.574306 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 43) took 3353 ms. Trace:
I20250905 19:52:32.576475 2376 rpcz_store.cc:276] 0905 19:52:29.220765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.220911 (+ 146us) service_pool.cc:225] Handling call
0905 19:52:32.506226 (+3285315us) ts_tablet_manager.cc:1145] Deleting tablet 42f3c1efb94e446fb0128b6b412821e2
0905 19:52:32.506238 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.532676 (+ 26438us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.546638 (+ 13962us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.573939 (+ 27301us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7517,"tablet-delete.queue_time_us":3224630}
I20250905 19:52:32.580675 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 42f3c1efb94e446fb0128b6b412821e2 (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]) successfully deleted
I20250905 19:52:32.582118 2376 tablet_replica.cc:331] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.582947 1169 tablet_bootstrap.cc:654] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.585698 2376 raft_consensus.cc:2241] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
W20250905 19:52:32.590987 848 consensus_peers.cc:489] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.592887 1171 tablet_bootstrap.cc:492] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:32.593276 849 consensus_peers.cc:489] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: fd793bce71e344f18bb177a53ab1bfc1. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.593554 1171 ts_tablet_manager.cc:1397] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.113s user 0.007s sys 0.009s
I20250905 19:52:32.593917 1543 catalog_manager.cc:5582] T 4dd2a96b8180438fbc722be1c18a0189 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.594902 1170 tablet_bootstrap.cc:654] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.595669 1171 raft_consensus.cc:357] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.596145 1171 raft_consensus.cc:383] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.596290 1171 raft_consensus.cc:738] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.596493 2379 ts_tablet_manager.cc:1939] T ab4f69c45295422f85b310aa55220e9b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.596827 1171 consensus_queue.cc:260] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.597343 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 67) took 3230 ms. Trace:
I20250905 19:52:32.597440 2379 rpcz_store.cc:276] 0905 19:52:29.366669 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.366757 (+ 88us) service_pool.cc:225] Handling call
0905 19:52:32.553913 (+3187156us) ts_tablet_manager.cc:1145] Deleting tablet ab4f69c45295422f85b310aa55220e9b
0905 19:52:32.557863 (+ 3950us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.564371 (+ 6508us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.567387 (+ 3016us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.597319 (+ 29932us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3178204}
I20250905 19:52:32.597872 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 80) took 3224 ms. Trace:
I20250905 19:52:32.597952 2379 rpcz_store.cc:276] 0905 19:52:29.372949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.373018 (+ 69us) service_pool.cc:225] Handling call
0905 19:52:32.597732 (+3224714us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:32.597742 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.597853 (+ 111us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3221392}
I20250905 19:52:32.592666 2376 raft_consensus.cc:2270] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.598217 2379 tablet_replica.cc:331] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.600389 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ab4f69c45295422f85b310aa55220e9b (table default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]) successfully deleted
I20250905 19:52:32.598402 2379 raft_consensus.cc:2241] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:32.600718 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3de936d2309a411eabbd59f7b73ce7f3 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3de936d2309a411eabbd59f7b73ce7f3 already in progress: creating tablet
I20250905 19:52:32.600857 2379 raft_consensus.cc:2270] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.603619 1170 tablet_bootstrap.cc:492] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.603763 1170 ts_tablet_manager.cc:1397] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.000s
I20250905 19:52:32.603375 1171 ts_tablet_manager.cc:1428] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250905 19:52:32.605175 2577 raft_consensus.cc:491] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.605304 2577 raft_consensus.cc:513] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.605901 2577 leader_election.cc:290] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.606760 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a1b4989f17454aa58c05727b797c2009" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.607239 849 leader_election.cc:343] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a1b4989f17454aa58c05727b797c2009
W20250905 19:52:32.607470 848 leader_election.cc:343] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.607589 848 leader_election.cc:304] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.608067 2577 raft_consensus.cc:2747] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.607237 2376 ts_tablet_manager.cc:1905] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.608321 1171 tablet_bootstrap.cc:492] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.610448 1170 raft_consensus.cc:357] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.610623 1170 raft_consensus.cc:383] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.610673 1170 raft_consensus.cc:738] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.610903 1170 consensus_queue.cc:260] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.611411 1170 ts_tablet_manager.cc:1428] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.000s sys 0.000s
I20250905 19:52:32.611850 1170 tablet_bootstrap.cc:492] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.613804 1170 tablet_bootstrap.cc:654] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.614265 2577 raft_consensus.cc:491] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.614394 2577 raft_consensus.cc:513] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.615000 2379 ts_tablet_manager.cc:1905] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.615727 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.615932 1037 raft_consensus.cc:2466] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.616596 849 leader_election.cc:304] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.615047 2577 leader_election.cc:290] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.616897 2577 raft_consensus.cc:2802] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.616973 2577 raft_consensus.cc:491] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.617033 2577 raft_consensus.cc:3058] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.619506 2577 raft_consensus.cc:513] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.620728 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:32.620961 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.621131 1543 catalog_manager.cc:5582] T 31e332cdfedd4ff6b5026baa73e2caf3 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.621356 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.621529 1037 raft_consensus.cc:3058] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.621719 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:32.622740 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:32.623147 1543 catalog_manager.cc:5582] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.623512 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:32.624295 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
W20250905 19:52:32.625314 848 leader_election.cc:343] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.625962 2577 leader_election.cc:290] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.626700 1037 raft_consensus.cc:2466] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:32.626621 1543 catalog_manager.cc:5582] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:32.627108 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.627223 849 leader_election.cc:304] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:32.627804 848 leader_election.cc:343] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.628182 2577 raft_consensus.cc:2802] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.628435 2577 raft_consensus.cc:695] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.628821 2577 consensus_queue.cc:237] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.632622 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:32.634584 2379 ts_tablet_manager.cc:1918] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.635329 1543 catalog_manager.cc:5582] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:32.636296 2652 ts_tablet_manager.cc:1918] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.638880 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:32.640028 2652 log.cc:1199] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:32.640089 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:32.640588 2652 ts_tablet_manager.cc:1939] T a2ff206366ad4344b4df5fc6a5584f8a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.640683 2379 log.cc:1199] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/2683a36998674f739b41d2a57339954c
I20250905 19:52:32.641101 2379 ts_tablet_manager.cc:1939] T 2683a36998674f739b41d2a57339954c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.641913 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 51) took 1127 ms. Trace:
I20250905 19:52:32.641961 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 81) took 3265 ms. Trace:
I20250905 19:52:32.641680 1171 tablet_bootstrap.cc:654] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.642050 2379 rpcz_store.cc:276] 0905 19:52:29.376256 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.377523 (+ 1267us) service_pool.cc:225] Handling call
0905 19:52:32.598174 (+3220651us) ts_tablet_manager.cc:1145] Deleting tablet 2683a36998674f739b41d2a57339954c
0905 19:52:32.598182 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.618151 (+ 19969us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.634466 (+ 16315us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.641939 (+ 7473us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3220481}
I20250905 19:52:32.642395 2379 tablet_replica.cc:331] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.642606 2379 raft_consensus.cc:2241] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.642958 2379 raft_consensus.cc:2270] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.643543 2379 ts_tablet_manager.cc:1905] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.643822 712 consensus_peers.cc:489] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.642637 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 2683a36998674f739b41d2a57339954c (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]) successfully deleted
I20250905 19:52:32.644932 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:32.645085 2652 rpcz_store.cc:276] 0905 19:52:31.514488 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.515792 (+ 1304us) service_pool.cc:225] Handling call
0905 19:52:32.496814 (+981022us) ts_tablet_manager.cc:1145] Deleting tablet a2ff206366ad4344b4df5fc6a5584f8a
0905 19:52:32.496824 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.519601 (+ 22777us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.522634 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.641893 (+119259us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":978219}
I20250905 19:52:32.645716 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a2ff206366ad4344b4df5fc6a5584f8a (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:32.645892 1543 catalog_manager.cc:5582] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.646112 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 57) took 1130 ms. Trace:
I20250905 19:52:32.646786 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:32.647132 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:32.650876 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:32.652282 1169 tablet_bootstrap.cc:492] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.652487 1169 ts_tablet_manager.cc:1397] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.126s user 0.001s sys 0.002s
I20250905 19:52:32.653294 1169 raft_consensus.cc:357] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.653522 1169 raft_consensus.cc:383] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.653620 1169 raft_consensus.cc:738] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.653867 1169 consensus_queue.cc:260] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.654469 1169 ts_tablet_manager.cc:1428] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250905 19:52:32.656457 986 consensus_peers.cc:489] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 06850faa25de4b3d83f6bf98ccd370f0. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.657224 1169 tablet_bootstrap.cc:492] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.647924 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:32.660918 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:32.646735 2652 rpcz_store.cc:276] 0905 19:52:31.515143 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.515928 (+ 785us) service_pool.cc:225] Handling call
0905 19:52:32.646007 (+1130079us) ts_tablet_manager.cc:1145] Deleting tablet 813f208881ac4782874138987a710d93
0905 19:52:32.646018 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.646095 (+ 77us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1127328}
W20250905 19:52:32.662446 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 813f208881ac4782874138987a710d93 because the tablet was not found. No further retry: Not found: Tablet not found: 813f208881ac4782874138987a710d93
I20250905 19:52:32.662848 2652 tablet_replica.cc:331] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.665395 1170 tablet_bootstrap.cc:492] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.665553 1170 ts_tablet_manager.cc:1397] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.054s user 0.011s sys 0.000s
I20250905 19:52:32.665743 2652 raft_consensus.cc:2241] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.665957 2652 raft_consensus.cc:2270] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.666389 1170 raft_consensus.cc:357] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.666574 1170 raft_consensus.cc:383] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.666643 1170 raft_consensus.cc:738] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.666862 1170 consensus_queue.cc:260] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.667380 1170 ts_tablet_manager.cc:1428] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:32.667874 2652 ts_tablet_manager.cc:1905] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.668320 1543 catalog_manager.cc:5582] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.668923 1170 tablet_bootstrap.cc:492] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.668001 1169 tablet_bootstrap.cc:654] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.670756 1170 tablet_bootstrap.cc:654] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.672183 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:32.673453 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:32.676657 2379 ts_tablet_manager.cc:1918] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.676937 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:32.677788 1543 catalog_manager.cc:5582] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:32.678267 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:32.678725 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:32.679179 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:32.679875 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:32.683076 1171 tablet_bootstrap.cc:492] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.683421 1171 ts_tablet_manager.cc:1397] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.005s
I20250905 19:52:32.683517 2379 log.cc:1199] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:32.684017 2379 ts_tablet_manager.cc:1939] T f28e403dc2f6471ebea1805fd843cfb1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.684885 1543 catalog_manager.cc:5582] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.683385 1170 tablet_bootstrap.cc:492] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.685348 1170 ts_tablet_manager.cc:1397] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20250905 19:52:32.686014 1171 raft_consensus.cc:357] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.686443 1171 raft_consensus.cc:383] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.686499 1170 raft_consensus.cc:357] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.686703 1170 raft_consensus.cc:383] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.686648 1171 raft_consensus.cc:738] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.686784 1170 raft_consensus.cc:738] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.686978 1849 raft_consensus.cc:491] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.687048 1170 consensus_queue.cc:260] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.687122 1849 raft_consensus.cc:513] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.687220 1171 consensus_queue.cc:260] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.687691 1170 ts_tablet_manager.cc:1428] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:32.687820 1849 leader_election.cc:290] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.688087 1170 tablet_bootstrap.cc:492] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.688627 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.688606 1171 ts_tablet_manager.cc:1428] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.001s
I20250905 19:52:32.688988 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 70) took 3318 ms. Trace:
I20250905 19:52:32.689070 2379 rpcz_store.cc:276] 0905 19:52:29.370679 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371726 (+ 1047us) service_pool.cc:225] Handling call
0905 19:52:32.642347 (+3270621us) ts_tablet_manager.cc:1145] Deleting tablet f28e403dc2f6471ebea1805fd843cfb1
0905 19:52:32.642360 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.659743 (+ 17383us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.676556 (+ 16813us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.688971 (+ 12415us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3265840}
I20250905 19:52:32.691160 2160 raft_consensus.cc:491] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.691293 2160 raft_consensus.cc:513] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.691900 2160 leader_election.cc:290] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.689517 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet f28e403dc2f6471ebea1805fd843cfb1 (table default.Table_332f3b495a6548caa4cc12a01588ee6d [id=f65bebcb5eee4ed6bf4f15339a018969]) successfully deleted
I20250905 19:52:32.692332 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:32.692603 2379 tablet_replica.cc:331] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.692859 2379 raft_consensus.cc:2241] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.693779 1543 catalog_manager.cc:5582] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
W20250905 19:52:32.693068 711 leader_election.cc:343] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: d296428b70f9411190b6c408d81639f2
I20250905 19:52:32.694353 2379 raft_consensus.cc:2270] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.695072 2379 ts_tablet_manager.cc:1905] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.697096 849 consensus_peers.cc:489] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d296428b70f9411190b6c408d81639f2. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.698892 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:32.699236 1543 catalog_manager.cc:5582] T a3a50526bb604a218649b24cbf65a65e P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.700313 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:32.703778 1543 catalog_manager.cc:5582] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.704504 1543 catalog_manager.cc:5582] T 8092952d755848ccaba3a2ad94b66dad P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.705072 1543 catalog_manager.cc:5582] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:32.705451 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:32.705916 1543 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:32.708791 2379 ts_tablet_manager.cc:1918] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.709319 1532 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9] alter complete (version 1)
I20250905 19:52:32.711616 2652 ts_tablet_manager.cc:1918] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.711714 2652 log.cc:1199] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/f48f9f789b6e46ec871a0fd007770dfc
I20250905 19:52:32.714655 1517 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b] alter complete (version 5)
I20250905 19:52:32.716125 2652 ts_tablet_manager.cc:1939] T f48f9f789b6e46ec871a0fd007770dfc P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.716442 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 55) took 1201 ms. Trace:
I20250905 19:52:32.716511 2652 rpcz_store.cc:276] 0905 19:52:31.514924 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.515905 (+ 981us) service_pool.cc:225] Handling call
0905 19:52:32.662796 (+1146891us) ts_tablet_manager.cc:1145] Deleting tablet f48f9f789b6e46ec871a0fd007770dfc
0905 19:52:32.662806 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.693929 (+ 31123us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.711506 (+ 17577us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.716428 (+ 4922us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1143909}
I20250905 19:52:32.716849 2652 tablet_replica.cc:331] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.717042 2652 raft_consensus.cc:2241] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.717038 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet f48f9f789b6e46ec871a0fd007770dfc (table default.Table_2af22804e18e430ea738862d0bade226 [id=82e44ec5bc0e43e2ab11552c4127f225]) successfully deleted
I20250905 19:52:32.717193 2652 raft_consensus.cc:2270] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.717712 2652 ts_tablet_manager.cc:1905] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.718001 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3587780819704008b7f4ee6abc714db6" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.718359 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d296428b70f9411190b6c408d81639f2" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.718505 1171 tablet_bootstrap.cc:492] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:32.722736 1171 tablet_bootstrap.cc:654] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.723075 1517 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f] alter complete (version 3)
I20250905 19:52:32.724107 2577 raft_consensus.cc:491] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.724270 2577 raft_consensus.cc:513] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.725090 2577 leader_election.cc:290] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.725492 2684 raft_consensus.cc:491] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.725606 2684 raft_consensus.cc:513] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.726277 2684 leader_election.cc:290] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.726918 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.727463 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f34bdf651d1412690039dc6ea1095d0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.727612 1037 raft_consensus.cc:2466] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:32.729406 986 leader_election.cc:343] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.730012 849 leader_election.cc:304] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.730823 2577 raft_consensus.cc:2802] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.730901 2577 raft_consensus.cc:491] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.730955 2577 raft_consensus.cc:3058] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.734931 2568 raft_consensus.cc:491] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.735060 2568 raft_consensus.cc:513] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.735608 2568 leader_election.cc:290] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.736415 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b44c00d3b54af6b0752575e6898839" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.736603 1037 raft_consensus.cc:2466] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.737483 1522 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7] alter complete (version 2)
I20250905 19:52:32.737529 1540 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07] alter complete (version 2)
I20250905 19:52:32.739326 849 leader_election.cc:304] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.739666 2568 raft_consensus.cc:2802] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.739739 2568 raft_consensus.cc:491] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.739807 2568 raft_consensus.cc:3058] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.743247 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.744079 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.748162 2652 ts_tablet_manager.cc:1918] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.749421 2568 raft_consensus.cc:513] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.750725 2652 log.cc:1199] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/cbfd03ad0bd34b6ca86e7b07a62b2a8e
I20250905 19:52:32.750926 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.751367 2652 ts_tablet_manager.cc:1939] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.752310 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.753234 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 50) took 1238 ms. Trace:
I20250905 19:52:32.753705 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 2350bd08f977424ba94ee8fa01f66efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.753235 2577 raft_consensus.cc:513] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.753588 2652 rpcz_store.cc:276] 0905 19:52:31.514398 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.515765 (+ 1367us) service_pool.cc:225] Handling call
0905 19:52:32.716793 (+1201028us) ts_tablet_manager.cc:1145] Deleting tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
0905 19:52:32.716805 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.728909 (+ 12104us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.748043 (+ 19134us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.753215 (+ 5172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":436,"tablet-delete.queue_time_us":1198266}
I20250905 19:52:32.754328 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e (table default.Table_c7c4cd0bb1b14ead83f2942518f4c30f [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
I20250905 19:52:32.754693 2652 tablet_replica.cc:331] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.755324 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.755632 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.755877 2652 raft_consensus.cc:2241] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.756446 2652 raft_consensus.cc:2270] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.756740 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f34bdf651d1412690039dc6ea1095d0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.756908 1037 raft_consensus.cc:3058] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.758038 1522 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e] alter complete (version 2)
I20250905 19:52:32.759572 2376 ts_tablet_manager.cc:1918] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.757238 2652 ts_tablet_manager.cc:1905] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.758009 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.758422 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.760008 2685 raft_consensus.cc:491] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.758836 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.759958 2376 log.cc:1199] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a8b6823580cb4040b49ed0a49b748965
I20250905 19:52:32.759191 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.759308 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:32.760119 2685 raft_consensus.cc:513] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.760731 2685 leader_election.cc:290] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.762169 2376 ts_tablet_manager.cc:1939] T a8b6823580cb4040b49ed0a49b748965 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:32.762171 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9d9815864aa7bf5ca9a5c4485390" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:32.763054 986 leader_election.cc:343] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.763060 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 44) took 3498 ms. Trace:
I20250905 19:52:32.763283 2376 rpcz_store.cc:276] 0905 19:52:29.264253 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.272005 (+ 7752us) service_pool.cc:225] Handling call
0905 19:52:32.581422 (+3309417us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:32.581434 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.673489 (+ 92055us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.703093 (+ 29604us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.762998 (+ 59905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3293085}
I20250905 19:52:32.764698 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a8b6823580cb4040b49ed0a49b748965 (table default.Table_aa894f644dbe444bbe04d396b9ceb056 [id=4172afda1ef0438cb4f4695bb8340e66]) successfully deleted
W20250905 19:52:32.764842 1547 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 68) took 3233 ms (client timeout 1000 ms). Trace:
W20250905 19:52:32.764955 1547 rpcz_store.cc:269] 0905 19:52:29.530999 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.572249 (+ 41250us) service_pool.cc:225] Handling call
0905 19:52:32.764794 (+3192545us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":42150,"reported_tablets":217}
I20250905 19:52:32.765209 2376 tablet_replica.cc:331] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:32.765640 2376 raft_consensus.cc:2241] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.766320 2376 raft_consensus.cc:2270] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.766644 2413 raft_consensus.cc:491] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.766785 2413 raft_consensus.cc:513] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.767357 2413 leader_election.cc:290] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.767470 2413 consensus_queue.cc:1035] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:32.768271 1542 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet d495b6ba92e84b588b4780c2896195c0 consensus state has no leader
I20250905 19:52:32.768451 1542 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 50) took 4718 ms. Trace:
I20250905 19:52:32.759698 1037 raft_consensus.cc:2466] T 7f34bdf651d1412690039dc6ea1095d0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:32.768991 2568 leader_election.cc:290] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.768576 1542 rpcz_store.cc:276] 0905 19:52:28.050007 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.822852 (+3772845us) service_pool.cc:225] Handling call
0905 19:52:31.823361 (+ 509us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:31.823621 (+ 260us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:31.823835 (+ 214us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:31.823939 (+ 104us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:31.844380 (+ 20441us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:32.768419 (+924039us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:31.832105 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:31.832255 (+ 150us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:31.832261 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:31.832263 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:31.832395 (+ 132us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:31.832408 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:31.832413 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:31.832476 (+ 63us) tablet.cc:699] Row locks acquired
0905 19:52:31.832479 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:31.832516 (+ 37us) write_op.cc:270] Start()
0905 19:52:31.832551 (+ 35us) write_op.cc:276] Timestamp: P: 1757101951832500 usec, L: 0
0905 19:52:31.832561 (+ 10us) op_driver.cc:348] REPLICATION: starting
0905 19:52:31.832787 (+ 226us) log.cc:844] Serialized 405 byte log entry
0905 19:52:31.836346 (+ 3559us) op_driver.cc:464] REPLICATION: finished
0905 19:52:31.843811 (+ 7465us) write_op.cc:301] APPLY: starting
0905 19:52:31.843829 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:31.843872 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:31.843915 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:31.843987 (+ 72us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:31.844020 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:31.844027 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:31.844110 (+ 83us) log.cc:844] Serialized 23 byte log entry
0905 19:52:31.844170 (+ 60us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.844191 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:31.844198 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:31.844265 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.844268 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":7385,"apply.run_cpu_time_us":465,"apply.run_wall_time_us":466,"num_ops":1,"prepare.queue_time_us":7598,"prepare.run_cpu_time_us":778,"prepare.run_wall_time_us":778,"raft.queue_time_us":796,"raft.run_cpu_time_us":329,"raft.run_wall_time_us":1618,"replication_time_us":3756}]]}
I20250905 19:52:32.769120 2577 leader_election.cc:290] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.775622 2652 ts_tablet_manager.cc:1918] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.775820 2652 log.cc:1199] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a8b6823580cb4040b49ed0a49b748965
I20250905 19:52:32.776355 2652 ts_tablet_manager.cc:1939] T a8b6823580cb4040b49ed0a49b748965 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.777225 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 63) took 1257 ms. Trace:
I20250905 19:52:32.778106 1169 tablet_bootstrap.cc:492] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.778292 1169 ts_tablet_manager.cc:1397] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.121s user 0.018s sys 0.005s
I20250905 19:52:32.779502 1169 raft_consensus.cc:357] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.779717 1169 raft_consensus.cc:383] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.779803 1169 raft_consensus.cc:738] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.780079 1169 consensus_queue.cc:260] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.784947 2376 ts_tablet_manager.cc:1905] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.786439 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4b31443b0f840749e12435e197a204a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.786725 1037 raft_consensus.cc:2466] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.788684 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b44c00d3b54af6b0752575e6898839" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.788861 1051 raft_consensus.cc:3058] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.791564 1051 raft_consensus.cc:2466] T d6b44c00d3b54af6b0752575e6898839 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:32.791890 1540 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 64) took 4241 ms (client timeout 1000 ms). Trace:
I20250905 19:52:32.792420 1169 ts_tablet_manager.cc:1428] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250905 19:52:32.792752 1169 tablet_bootstrap.cc:492] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:32.794584 1169 tablet_bootstrap.cc:654] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.777351 2652 rpcz_store.cc:276] 0905 19:52:31.519512 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.519566 (+ 54us) service_pool.cc:225] Handling call
0905 19:52:32.754641 (+1235075us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:32.754652 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.767381 (+ 12729us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.775460 (+ 8079us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.777205 (+ 1745us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1234872}
I20250905 19:52:32.796110 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a8b6823580cb4040b49ed0a49b748965 (table default.Table_aa894f644dbe444bbe04d396b9ceb056 [id=4172afda1ef0438cb4f4695bb8340e66]) successfully deleted
I20250905 19:52:32.796471 2652 tablet_replica.cc:331] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.796703 2652 raft_consensus.cc:2241] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.796905 2652 raft_consensus.cc:2270] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.797554 2652 ts_tablet_manager.cc:1905] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.804389 849 leader_election.cc:304] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.805027 2413 raft_consensus.cc:2802] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.805186 2413 raft_consensus.cc:695] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.805478 2413 consensus_queue.cc:237] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.808717 2684 raft_consensus.cc:491] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.808879 2684 raft_consensus.cc:513] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.809597 2684 leader_election.cc:290] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.810232 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:32.810812 986 leader_election.cc:343] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.813156 849 leader_election.cc:304] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.813529 2568 raft_consensus.cc:2802] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.813632 2568 raft_consensus.cc:491] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.813706 2568 raft_consensus.cc:3058] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.814407 2652 ts_tablet_manager.cc:1918] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.814929 2652 log.cc:1199] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/44769f1170354870ae2d34fdb67ffbae
I20250905 19:52:32.815362 2652 ts_tablet_manager.cc:1939] T 44769f1170354870ae2d34fdb67ffbae P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:32.816689 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 62) took 1297 ms. Trace:
I20250905 19:52:32.817338 2652 rpcz_store.cc:276] 0905 19:52:31.519369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.519856 (+ 487us) service_pool.cc:225] Handling call
0905 19:52:32.796420 (+1276564us) ts_tablet_manager.cc:1145] Deleting tablet 44769f1170354870ae2d34fdb67ffbae
0905 19:52:32.796431 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.800734 (+ 4303us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.814290 (+ 13556us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.816667 (+ 2377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1276389}
I20250905 19:52:32.818056 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 44769f1170354870ae2d34fdb67ffbae (table default.Table_885d880838c44eab9ff1220768e59373 [id=3235ea3e87b249e884b027b840d4fd77]) successfully deleted
I20250905 19:52:32.818526 2652 tablet_replica.cc:331] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:32.818950 2652 raft_consensus.cc:2241] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.819367 2652 raft_consensus.cc:2270] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.819941 2379 log.cc:1199] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:32.718796 712 leader_election.cc:304] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:32.820389 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 101634us
I20250905 19:52:32.820461 2379 ts_tablet_manager.cc:1939] T 9630f1feb6d74472b5f6f4b54151311c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.820830 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 87) took 3443 ms. Trace:
I20250905 19:52:32.820945 2379 rpcz_store.cc:276] 0905 19:52:29.376894 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.377714 (+ 820us) service_pool.cc:225] Handling call
0905 19:52:32.692479 (+3314765us) ts_tablet_manager.cc:1145] Deleting tablet 9630f1feb6d74472b5f6f4b54151311c
0905 19:52:32.692492 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.698127 (+ 5635us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.708667 (+ 10540us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.820785 (+112118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3310657}
I20250905 19:52:32.821031 849 leader_election.cc:304] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.821280 2379 tablet_replica.cc:331] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.821267 2577 raft_consensus.cc:2802] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.821374 1171 tablet_bootstrap.cc:492] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:32.821528 2379 raft_consensus.cc:2241] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.821591 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 01c20eea1b024d7f93f18f0c089efc42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.821712 1171 ts_tablet_manager.cc:1397] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.103s user 0.005s sys 0.003s
I20250905 19:52:32.821879 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.821925 2379 raft_consensus.cc:2270] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.821743 2577 raft_consensus.cc:695] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.822043 712 leader_election.cc:304] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:32.822266 2577 consensus_queue.cc:237] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.822646 1849 raft_consensus.cc:2747] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.822818 1849 raft_consensus.cc:2747] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.823032 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.823239 740 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.823427 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.824086 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.823819 1171 raft_consensus.cc:357] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.824277 737 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.824239 1171 raft_consensus.cc:383] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.824452 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.824378 1171 raft_consensus.cc:738] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.824664 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.824853 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.825028 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:32.824920 1171 consensus_queue.cc:260] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.825508 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9630f1feb6d74472b5f6f4b54151311c (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]) successfully deleted
I20250905 19:52:32.728427 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.729208 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f34bdf651d1412690039dc6ea1095d0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.737978 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b44c00d3b54af6b0752575e6898839" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.751008 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d6b44c00d3b54af6b0752575e6898839" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.754983 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7f34bdf651d1412690039dc6ea1095d0" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:32.761615 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9d9815864aa7bf5ca9a5c4485390" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.768656 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4b31443b0f840749e12435e197a204a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.780937 1170 tablet_bootstrap.cc:654] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:32.826514 984 leader_election.cc:343] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.826443 1171 ts_tablet_manager.cc:1428] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 19:52:32.826647 984 leader_election.cc:304] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.788106 2160 raft_consensus.cc:491] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.811616 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.827435 2160 raft_consensus.cc:513] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.828202 2160 leader_election.cc:290] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:32.829268 848 leader_election.cc:343] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:32.829661 848 leader_election.cc:343] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:32.830066 848 leader_election.cc:343] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:32.830324 848 leader_election.cc:343] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:32.830714 848 leader_election.cc:343] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:32.831043 848 consensus_peers.cc:489] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 54628965b82e470d811eff95ffc56c5f. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:32.832044 984 leader_election.cc:343] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.832154 984 leader_election.cc:304] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.832384 2685 raft_consensus.cc:2747] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.832525 2685 raft_consensus.cc:2747] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:32.832564 984 leader_election.cc:343] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.832639 984 leader_election.cc:304] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.832815 2685 raft_consensus.cc:2747] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:32.792824 1540 rpcz_store.cc:269] 0905 19:52:28.550559 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.131281 (+580722us) service_pool.cc:225] Handling call
0905 19:52:29.747612 (+616331us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.747625 (+ 13us) write_op.cc:454] Released schema lock
0905 19:52:32.791761 (+3044136us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:29.626826 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:29.626969 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:29.626977 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:29.626979 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:29.627106 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:29.627127 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:29.627136 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:29.627227 (+ 91us) tablet.cc:699] Row locks acquired
0905 19:52:29.627232 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:29.627281 (+ 49us) write_op.cc:270] Start()
0905 19:52:29.627331 (+ 50us) write_op.cc:276] Timestamp: P: 1757101949627260 usec, L: 0
0905 19:52:29.627338 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:29.627643 (+ 305us) log.cc:844] Serialized 356 byte log entry
0905 19:52:29.631577 (+ 3934us) op_driver.cc:464] REPLICATION: finished
0905 19:52:29.632296 (+ 719us) write_op.cc:301] APPLY: starting
0905 19:52:29.632316 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:29.632364 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:29.632404 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:29.632464 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:29.632509 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:29.632521 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:29.632604 (+ 83us) log.cc:844] Serialized 23 byte log entry
0905 19:52:29.632667 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:29.632690 (+ 23us) write_op.cc:454] Released schema lock
0905 19:52:29.632698 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:31.698155 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:31.698314 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:31.698324 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:31.698328 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:31.698470 (+ 142us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:31.698484 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:31.698491 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:31.698603 (+ 112us) tablet.cc:699] Row locks acquired
0905 19:52:31.698606 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:31.698649 (+ 43us) write_op.cc:270] Start()
0905 19:52:31.698690 (+ 41us) write_op.cc:276] Timestamp: P: 1757101951698631 usec, L: 0
0905 19:52:31.698693 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:31.698919 (+ 226us) log.cc:844] Serialized 356 byte log entry
0905 19:52:31.797860 (+ 98941us) op_driver.cc:464] REPLICATION: finished
0905 19:52:31.829323 (+ 31463us) write_op.cc:301] APPLY: starting
0905 19:52:31.829344 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:31.829389 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:31.829443 (+ 54us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:31.829510 (+ 67us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:31.829545 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:31.829552 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:31.829690 (+ 138us) log.cc:844] Serialized 23 byte log entry
0905 19:52:31.829754 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.829773 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:31.829781 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:31.829928 (+ 147us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.829933 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":167872,"reported_tablets":182,"child_traces":[["op",{"apply.queue_time_us":653,"apply.run_cpu_time_us":463,"apply.run_wall_time_us":463,"mutex_wait_us":86726,"num_ops":1,"prepare.queue_time_us":86817,"prepare.run_cpu_time_us":961,"prepare.run_wall_time_us":961,"raft.queue_time_us":584,"raft.run_cpu_time_us":377,"raft.run_wall_time_us":376,"replication_time_us":4202}],["op",{"apply.queue_time_us":31342,"apply.run_cpu_time_us":548,"apply.run_wall_time_us":610,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":889,"prepare.run_wall_time_us":889,"replication_time_us":99091,"thread_start_us":12995,"threads_started":1}]]}
I20250905 19:52:32.838573 2379 ts_tablet_manager.cc:1905] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:32.838977 711 leader_election.cc:343] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:32.842253 2160 raft_consensus.cc:491] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.842449 2160 raft_consensus.cc:513] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.843125 2160 leader_election.cc:290] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.842514 771 raft_consensus.cc:3058] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.846717 2684 raft_consensus.cc:491] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.846868 2684 raft_consensus.cc:513] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.847501 2684 leader_election.cc:290] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.847877 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.848320 984 leader_election.cc:343] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.849495 2379 ts_tablet_manager.cc:1918] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.849634 2379 log.cc:1199] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:32.849618 2684 raft_consensus.cc:491] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:32.849732 2684 raft_consensus.cc:513] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.850062 2379 ts_tablet_manager.cc:1939] T 03a12cf719f34096bb82f5846bbe3b2d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.850260 2684 leader_election.cc:290] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.850872 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 85) took 3474 ms. Trace:
W20250905 19:52:32.851023 984 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:32.850986 2379 rpcz_store.cc:276] 0905 19:52:29.376713 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.378005 (+ 1292us) service_pool.cc:225] Handling call
0905 19:52:32.821211 (+3443206us) ts_tablet_manager.cc:1145] Deleting tablet 03a12cf719f34096bb82f5846bbe3b2d
0905 19:52:32.821221 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.846249 (+ 25028us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.849367 (+ 3118us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.850810 (+ 1443us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3441670}
I20250905 19:52:32.851346 2379 tablet_replica.cc:331] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.851567 2379 raft_consensus.cc:2241] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:32.851752 2379 raft_consensus.cc:2270] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.852335 2379 ts_tablet_manager.cc:1905] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.856887 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.857666 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd793bce71e344f18bb177a53ab1bfc1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:32.858306 986 leader_election.cc:343] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.858418 986 leader_election.cc:304] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:32.858772 986 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:32.858850 986 leader_election.cc:304] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.859018 2685 raft_consensus.cc:2747] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.859158 2685 raft_consensus.cc:2747] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:32.860630 1170 tablet_bootstrap.cc:492] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.860831 1170 ts_tablet_manager.cc:1397] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.173s user 0.009s sys 0.002s
I20250905 19:52:32.861685 1170 raft_consensus.cc:357] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.861901 1170 raft_consensus.cc:383] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.861971 1170 raft_consensus.cc:738] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.862017 2565 raft_consensus.cc:491] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.862136 2565 raft_consensus.cc:513] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.862198 1170 consensus_queue.cc:260] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.862644 2565 leader_election.cc:290] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.862709 1170 ts_tablet_manager.cc:1428] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:32.863217 1170 tablet_bootstrap.cc:492] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:32.863543 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.863725 1037 raft_consensus.cc:2466] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:32.864079 1171 tablet_bootstrap.cc:492] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:32.864620 712 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:32.864867 712 leader_election.cc:304] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:32.865094 1170 tablet_bootstrap.cc:654] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.865963 2160 raft_consensus.cc:2747] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.866226 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c0ba1d4a7cd444c688d5a9bc8fd1a0ff" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.867591 849 leader_election.cc:304] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:32.870102 711 leader_election.cc:304] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:32.870378 711 raft_consensus.cc:2684] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:32.870671 2577 raft_consensus.cc:2802] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:32.870831 2577 raft_consensus.cc:491] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:32.871228 2577 raft_consensus.cc:3058] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.871670 768 raft_consensus.cc:3058] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.874081 2577 raft_consensus.cc:513] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.874295 2379 ts_tablet_manager.cc:1918] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:32.874423 2379 log.cc:1199] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:32.874866 2379 ts_tablet_manager.cc:1939] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.874862 2577 leader_election.cc:290] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.874994 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.875531 775 raft_consensus.cc:3058] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.875623 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 76) took 3504 ms. Trace:
I20250905 19:52:32.875766 2379 rpcz_store.cc:276] 0905 19:52:29.371190 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371649 (+ 459us) service_pool.cc:225] Handling call
0905 19:52:32.851262 (+3479613us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:32.851271 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.871113 (+ 19842us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.874146 (+ 3033us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.875581 (+ 1435us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3471315}
W20250905 19:52:32.879715 848 consensus_peers.cc:489] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.881726 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:32.882289 1037 raft_consensus.cc:3058] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.884238 2379 tablet_replica.cc:331] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.884349 2684 raft_consensus.cc:491] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.884502 2379 raft_consensus.cc:2241] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.884470 2684 raft_consensus.cc:513] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.885021 2684 leader_election.cc:290] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.885775 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 03a12cf719f34096bb82f5846bbe3b2d (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]) successfully deleted
I20250905 19:52:32.885874 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:32.886335 986 leader_election.cc:343] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.886682 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff (table default.Table_1c6571236cd7434d98aec149d06dc6b8 [id=6709b02c192c4ff88715b1f3b886b222]) successfully deleted
W20250905 19:52:32.891136 848 leader_election.cc:343] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.892189 2565 consensus_queue.cc:1035] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:32.869683 1171 tablet_bootstrap.cc:654] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.908604 2685 raft_consensus.cc:491] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:32.908777 2685 raft_consensus.cc:513] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.909442 2685 leader_election.cc:290] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:32.911224 986 leader_election.cc:343] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: a5af8232d31240ff855b79dcea29396b
I20250905 19:52:32.911747 2684 raft_consensus.cc:491] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.911865 2684 raft_consensus.cc:513] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.912482 2684 leader_election.cc:290] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:32.913432 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:32.913928 986 leader_election.cc:343] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.917678 2652 ts_tablet_manager.cc:1905] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.918656 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.919207 2379 raft_consensus.cc:2270] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.919718 771 raft_consensus.cc:3058] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:32.923307 2379 ts_tablet_manager.cc:1905] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.923413 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:32.923727 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:32.924820 848 leader_election.cc:343] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.930024 2565 consensus_queue.cc:1035] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:32.933722 2379 ts_tablet_manager.cc:1918] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.933679 2565 consensus_queue.cc:1035] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:32.933923 2379 log.cc:1199] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/00ac56c449ba4b7b8dbd5e87120799d9
W20250905 19:52:32.922518 984 leader_election.cc:343] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.934237 984 leader_election.cc:304] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.934453 2379 ts_tablet_manager.cc:1939] T 00ac56c449ba4b7b8dbd5e87120799d9 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.934473 2685 raft_consensus.cc:2747] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:32.934762 984 leader_election.cc:343] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.934864 984 leader_election.cc:304] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.934891 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 86) took 3558 ms. Trace:
I20250905 19:52:32.935045 2685 raft_consensus.cc:2747] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:32.935010 2379 rpcz_store.cc:276] 0905 19:52:29.376807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.378029 (+ 1222us) service_pool.cc:225] Handling call
0905 19:52:32.884142 (+3506113us) ts_tablet_manager.cc:1145] Deleting tablet 00ac56c449ba4b7b8dbd5e87120799d9
0905 19:52:32.884155 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.929585 (+ 45430us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.933550 (+ 3965us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.934819 (+ 1269us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3502484}
W20250905 19:52:32.935199 984 leader_election.cc:343] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: a5af8232d31240ff855b79dcea29396b
I20250905 19:52:32.935285 984 leader_election.cc:304] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.935468 2379 tablet_replica.cc:331] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.935753 2379 raft_consensus.cc:2241] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.935917 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 00ac56c449ba4b7b8dbd5e87120799d9 (table default.Table_3fa477c60f534a63b2cd4e54fe7c5c40 [id=8ce99397ad9f43508a20590a0d175ad1]) successfully deleted
I20250905 19:52:32.936057 2496 raft_consensus.cc:2747] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:32.940402 2379 raft_consensus.cc:2270] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.941211 2379 ts_tablet_manager.cc:1905] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.943011 1169 tablet_bootstrap.cc:492] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:32.943189 1169 ts_tablet_manager.cc:1397] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.151s user 0.004s sys 0.010s
I20250905 19:52:32.944607 1169 raft_consensus.cc:357] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.946338 1170 tablet_bootstrap.cc:492] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:32.946696 1170 ts_tablet_manager.cc:1397] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.084s user 0.011s sys 0.044s
I20250905 19:52:32.951272 1170 raft_consensus.cc:357] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.952826 1170 raft_consensus.cc:383] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:32.953665 1170 raft_consensus.cc:738] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:32.955499 1170 consensus_queue.cc:260] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.958691 1170 ts_tablet_manager.cc:1428] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.012s user 0.002s sys 0.002s
I20250905 19:52:32.960265 1170 tablet_bootstrap.cc:492] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:32.965569 848 consensus_peers.cc:489] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.933346 1037 raft_consensus.cc:2466] T 0d93a328d4934e5294ec66f739eb240b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:32.968290 849 leader_election.cc:304] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:32.968770 2738 raft_consensus.cc:2802] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:32.968997 2738 raft_consensus.cc:695] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:32.969297 2738 consensus_queue.cc:237] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.970734 2160 raft_consensus.cc:491] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.970753 1170 tablet_bootstrap.cc:654] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:32.970928 2160 raft_consensus.cc:513] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.971619 2160 leader_election.cc:290] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.973030 2379 ts_tablet_manager.cc:1918] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.973685 2379 log.cc:1199] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:32.975046 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:32.975443 2379 ts_tablet_manager.cc:1939] T b13d282e1dc54d1faa2882c9b39d17cb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:32.977393 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 83) took 3600 ms. Trace:
I20250905 19:52:32.977648 2577 raft_consensus.cc:491] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:32.977804 2577 raft_consensus.cc:513] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:32.977690 2379 rpcz_store.cc:276] 0905 19:52:29.376480 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.378120 (+ 1640us) service_pool.cc:225] Handling call
0905 19:52:32.935362 (+3557242us) ts_tablet_manager.cc:1145] Deleting tablet b13d282e1dc54d1faa2882c9b39d17cb
0905 19:52:32.935373 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.955270 (+ 19897us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.972173 (+ 16903us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.977316 (+ 5143us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3553581}
I20250905 19:52:32.978530 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet b13d282e1dc54d1faa2882c9b39d17cb (table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]) successfully deleted
I20250905 19:52:32.978576 2379 tablet_replica.cc:331] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:32.979068 2379 raft_consensus.cc:2241] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:32.981065 2379 raft_consensus.cc:2270] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:32.981750 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ea6df8634a314ceabcf3fd004bf7e241" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:32.982988 2577 leader_election.cc:290] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:32.983299 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9193487b4ab74dafa76a386ddcaf674b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:32.983914 848 leader_election.cc:343] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:32.985706 2379 ts_tablet_manager.cc:1905] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:32.987217 2652 ts_tablet_manager.cc:1918] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:32.987361 2652 log.cc:1199] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/c46ea498d6a14d1ebdffea415cfc7114
I20250905 19:52:32.987823 2652 ts_tablet_manager.cc:1939] T c46ea498d6a14d1ebdffea415cfc7114 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:32.996513 848 consensus_peers.cc:489] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:32.998512 712 leader_election.cc:304] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:32.999186 2160 raft_consensus.cc:2747] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.000288 771 raft_consensus.cc:3058] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.004660 1171 tablet_bootstrap.cc:492] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.004837 1171 ts_tablet_manager.cc:1397] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.141s user 0.002s sys 0.003s
I20250905 19:52:33.005677 1171 raft_consensus.cc:357] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.005899 1171 raft_consensus.cc:383] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.005947 1171 raft_consensus.cc:738] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.006155 1171 consensus_queue.cc:260] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.006824 1171 ts_tablet_manager.cc:1428] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:33.007117 1171 tablet_bootstrap.cc:492] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.008757 1171 tablet_bootstrap.cc:654] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.007324 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9193487b4ab74dafa76a386ddcaf674b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.010218 1040 raft_consensus.cc:2466] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.010787 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 61) took 1491 ms. Trace:
I20250905 19:52:33.011317 2652 rpcz_store.cc:276] 0905 19:52:31.519266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.520116 (+ 850us) service_pool.cc:225] Handling call
0905 19:52:32.818477 (+1298361us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:32.818487 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.957955 (+139468us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:32.987076 (+ 29121us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.010773 (+ 23697us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1298220}
I20250905 19:52:33.011940 2652 tablet_replica.cc:331] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.012205 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet c46ea498d6a14d1ebdffea415cfc7114 (table default.Table_f542b700d5364bfeb6e1e0111834fab0 [id=27f7071a498d467aa9047b07dac76b64]) successfully deleted
I20250905 19:52:33.012475 2652 raft_consensus.cc:2241] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.012634 2652 raft_consensus.cc:2270] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.013621 849 leader_election.cc:304] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.016775 2738 raft_consensus.cc:2802] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.016930 2738 raft_consensus.cc:491] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.017045 2738 raft_consensus.cc:3058] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.017797 2376 ts_tablet_manager.cc:1918] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.017925 2376 log.cc:1199] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9e9994863c374354838aa0f817d0057e
I20250905 19:52:33.018321 2376 ts_tablet_manager.cc:1939] T 9e9994863c374354838aa0f817d0057e P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:33.018687 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 73) took 3703 ms. Trace:
I20250905 19:52:33.018795 2376 rpcz_store.cc:276] 0905 19:52:29.315547 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.322826 (+ 7279us) service_pool.cc:225] Handling call
0905 19:52:32.765078 (+3442252us) ts_tablet_manager.cc:1145] Deleting tablet 9e9994863c374354838aa0f817d0057e
0905 19:52:32.765127 (+ 49us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:32.831322 (+ 66195us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.017652 (+186330us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.018668 (+ 1016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3442073}
I20250905 19:52:33.019173 2376 tablet_replica.cc:331] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:33.019399 2376 raft_consensus.cc:2241] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.019380 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9e9994863c374354838aa0f817d0057e (table default.Table_56e29cf512764408b243635ae01a6ab0 [id=aad015c74608490985ee75fbd380f0e3]) successfully deleted
I20250905 19:52:33.019757 2376 raft_consensus.cc:2270] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:33.013322 984 consensus_peers.cc:489] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2683a36998674f739b41d2a57339954c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.020812 2652 ts_tablet_manager.cc:1905] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.021128 2738 raft_consensus.cc:513] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.022228 2738 leader_election.cc:290] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.022372 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:33.022500 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9193487b4ab74dafa76a386ddcaf674b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:33.023118 848 leader_election.cc:343] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.025138 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9193487b4ab74dafa76a386ddcaf674b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.025305 1040 raft_consensus.cc:3058] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.028165 1040 raft_consensus.cc:2466] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.031327 849 leader_election.cc:304] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.031955 2738 raft_consensus.cc:2802] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.032193 2738 raft_consensus.cc:695] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:33.032539 2738 consensus_queue.cc:237] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.039768 2568 raft_consensus.cc:513] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.040052 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:33.044734 775 raft_consensus.cc:3058] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:33.048731 986 consensus_peers.cc:489] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 98e5ca7bd3e84916a11e01afbf48171d. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.052348 1849 raft_consensus.cc:491] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.052479 1849 raft_consensus.cc:513] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.053086 1849 leader_election.cc:290] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.054215 1849 raft_consensus.cc:491] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.054226 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.054335 1849 raft_consensus.cc:513] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.054853 1849 leader_election.cc:290] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.054942 2496 raft_consensus.cc:491] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.055063 2496 raft_consensus.cc:513] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.055630 2496 leader_election.cc:290] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.057377 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bb79e3126074736a23b128a8e14fab6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.057829 984 leader_election.cc:343] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.058652 1170 tablet_bootstrap.cc:492] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.058634 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:33.058725 768 raft_consensus.cc:3058] T 3587780819704008b7f4ee6abc714db6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.058946 1170 ts_tablet_manager.cc:1397] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.099s user 0.017s sys 0.013s
I20250905 19:52:33.060230 1169 raft_consensus.cc:383] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.060627 1170 raft_consensus.cc:357] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.060962 1170 raft_consensus.cc:383] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.061057 1170 raft_consensus.cc:738] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.061458 1170 consensus_queue.cc:260] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.062251 1169 raft_consensus.cc:738] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.062544 1169 consensus_queue.cc:260] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:33.063278 711 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:33.064076 1170 ts_tablet_manager.cc:1428] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:33.065331 1170 tablet_bootstrap.cc:492] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.069201 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:33.070921 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
W20250905 19:52:33.072422 712 leader_election.cc:343] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
I20250905 19:52:33.072552 712 leader_election.cc:304] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.073032 1849 raft_consensus.cc:2747] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.073228 1170 tablet_bootstrap.cc:654] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.074311 2652 ts_tablet_manager.cc:1918] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.076671 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:33.077126 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:33.077732 2652 log.cc:1199] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8503f6918ffb44efa36214633c93e31d
I20250905 19:52:33.074548 1169 ts_tablet_manager.cc:1428] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.131s user 0.014s sys 0.001s
I20250905 19:52:33.078023 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:33.078383 1169 tablet_bootstrap.cc:492] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.080631 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:33.081501 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:33.081710 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:33.083634 2652 ts_tablet_manager.cc:1939] T 8503f6918ffb44efa36214633c93e31d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.087836 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 60) took 1568 ms. Trace:
I20250905 19:52:33.088388 2652 rpcz_store.cc:276] 0905 19:52:31.519111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.520317 (+ 1206us) service_pool.cc:225] Handling call
0905 19:52:33.011895 (+1491578us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:33.011905 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.030904 (+ 18999us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.074194 (+ 43290us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.087806 (+ 13612us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1491438}
I20250905 19:52:33.088814 2652 tablet_replica.cc:331] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.088982 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8503f6918ffb44efa36214633c93e31d (table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]) successfully deleted
I20250905 19:52:33.089035 2652 raft_consensus.cc:2241] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.089363 2652 raft_consensus.cc:2270] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.090060 2652 ts_tablet_manager.cc:1905] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.092875 1170 tablet_bootstrap.cc:492] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.088327 1169 tablet_bootstrap.cc:654] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.093189 1170 ts_tablet_manager.cc:1397] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.001s
I20250905 19:52:33.094794 1170 raft_consensus.cc:357] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.095213 1170 raft_consensus.cc:383] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.095438 1170 raft_consensus.cc:738] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.096009 1170 consensus_queue.cc:260] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.040428 2376 ts_tablet_manager.cc:1905] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.057049 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bb79e3126074736a23b128a8e14fab6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.098277 1170 ts_tablet_manager.cc:1428] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
W20250905 19:52:33.098614 986 leader_election.cc:343] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.098737 986 leader_election.cc:304] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.099107 2496 raft_consensus.cc:2747] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.099623 1170 tablet_bootstrap.cc:492] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.100668 2568 leader_election.cc:290] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.101085 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4b31443b0f840749e12435e197a204a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:33.101085 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4b31443b0f840749e12435e197a204a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.101267 1040 raft_consensus.cc:3058] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:33.101613 848 leader_election.cc:343] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.103093 2496 raft_consensus.cc:491] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:33.103219 2496 raft_consensus.cc:513] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.103744 2496 leader_election.cc:290] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.103920 1040 raft_consensus.cc:2466] T c4b31443b0f840749e12435e197a204a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.104357 849 leader_election.cc:304] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:33.105350 986 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:33.069087 1171 tablet_bootstrap.cc:492] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.105743 2577 raft_consensus.cc:2802] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.105863 2577 raft_consensus.cc:695] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:33.106071 2577 consensus_queue.cc:237] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.106251 1170 tablet_bootstrap.cc:654] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.070402 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "efb70327f55844879f89449a7116010f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.076323 2413 raft_consensus.cc:491] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.107229 712 leader_election.cc:304] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.107384 2413 raft_consensus.cc:513] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.107650 1849 raft_consensus.cc:2747] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.108511 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9db56577a26e407b97fc33158c2c466c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.108631 2413 leader_election.cc:290] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.105759 1171 ts_tablet_manager.cc:1397] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.099s user 0.003s sys 0.000s
W20250905 19:52:33.109015 848 leader_election.cc:343] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.109628 1171 raft_consensus.cc:357] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.109783 1171 raft_consensus.cc:383] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.109848 1171 raft_consensus.cc:738] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.110011 1171 consensus_queue.cc:260] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:33.111786 849 leader_election.cc:343] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9db56577a26e407b97fc33158c2c466c
I20250905 19:52:33.112116 849 leader_election.cc:304] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.112316 1171 ts_tablet_manager.cc:1428] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:33.112603 1169 tablet_bootstrap.cc:492] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.112617 1171 tablet_bootstrap.cc:492] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.112748 2413 raft_consensus.cc:2747] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.112787 1169 ts_tablet_manager.cc:1397] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.034s user 0.005s sys 0.001s
I20250905 19:52:33.113531 1169 raft_consensus.cc:357] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.113710 1169 raft_consensus.cc:383] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.113766 1169 raft_consensus.cc:738] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:33.113776 984 leader_election.cc:343] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:33.113904 984 leader_election.cc:304] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.113982 1169 consensus_queue.cc:260] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.114092 1171 tablet_bootstrap.cc:654] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.114240 2685 raft_consensus.cc:2747] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:33.114414 1169 ts_tablet_manager.cc:1428] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:33.114673 1169 tablet_bootstrap.cc:492] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.115885 1169 tablet_bootstrap.cc:654] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.117411 2652 ts_tablet_manager.cc:1918] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.117549 2652 log.cc:1199] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ab4f69c45295422f85b310aa55220e9b
I20250905 19:52:33.118221 2652 ts_tablet_manager.cc:1939] T ab4f69c45295422f85b310aa55220e9b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.118975 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 64) took 1563 ms. Trace:
I20250905 19:52:33.119099 2652 rpcz_store.cc:276] 0905 19:52:31.555919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.556333 (+ 414us) service_pool.cc:225] Handling call
0905 19:52:33.088756 (+1532423us) ts_tablet_manager.cc:1145] Deleting tablet ab4f69c45295422f85b310aa55220e9b
0905 19:52:33.088767 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.100614 (+ 11847us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.117307 (+ 16693us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.118957 (+ 1650us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1532226}
I20250905 19:52:33.119143 2379 ts_tablet_manager.cc:1918] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.119608 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ab4f69c45295422f85b310aa55220e9b (table default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]) successfully deleted
I20250905 19:52:33.119838 2379 log.cc:1199] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:33.120317 2376 ts_tablet_manager.cc:1918] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.120425 2376 log.cc:1199] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/723168b075874dfcb3eb50c5134c499e
I20250905 19:52:33.120848 2376 ts_tablet_manager.cc:1939] T 723168b075874dfcb3eb50c5134c499e P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:33.121182 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 71) took 3805 ms. Trace:
I20250905 19:52:33.121256 2376 rpcz_store.cc:276] 0905 19:52:29.315358 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.315590 (+ 232us) service_pool.cc:225] Handling call
0905 19:52:33.019127 (+3703537us) ts_tablet_manager.cc:1145] Deleting tablet 723168b075874dfcb3eb50c5134c499e
0905 19:52:33.019135 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.101272 (+ 82137us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.120202 (+ 18930us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.121164 (+ 962us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3694254}
I20250905 19:52:33.121634 2376 tablet_replica.cc:331] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:33.121814 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 723168b075874dfcb3eb50c5134c499e (table default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]) successfully deleted
I20250905 19:52:33.121860 2376 raft_consensus.cc:2241] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.122103 2496 raft_consensus.cc:491] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.122193 2376 raft_consensus.cc:2270] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.122251 2496 raft_consensus.cc:513] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.122206 2379 ts_tablet_manager.cc:1939] T 302053e1611b4e22a9a2e7b60ae68f27 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:33.122805 2652 tablet_replica.cc:331] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.122829 2376 ts_tablet_manager.cc:1905] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.122893 2496 leader_election.cc:290] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.123050 2652 raft_consensus.cc:2241] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.123418 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.124202 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:33.124733 1849 raft_consensus.cc:491] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.124892 1849 raft_consensus.cc:513] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.125492 1849 leader_election.cc:290] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:33.126252 711 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:33.127502 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 82) took 3750 ms. Trace:
I20250905 19:52:33.127820 771 raft_consensus.cc:3058] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.128080 2379 rpcz_store.cc:276] 0905 19:52:29.376377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.378048 (+ 1671us) service_pool.cc:225] Handling call
0905 19:52:32.978385 (+3600337us) ts_tablet_manager.cc:1145] Deleting tablet 302053e1611b4e22a9a2e7b60ae68f27
0905 19:52:32.978433 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.108084 (+129651us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.118606 (+ 10522us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.127172 (+ 8566us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3596438}
I20250905 19:52:33.130496 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 88) took 3750 ms. Trace:
W20250905 19:52:33.131597 712 leader_election.cc:343] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
I20250905 19:52:33.131697 712 leader_election.cc:304] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.130918 2379 rpcz_store.cc:276] 0905 19:52:29.379673 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.380858 (+ 1185us) service_pool.cc:225] Handling call
0905 19:52:33.129687 (+3748829us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:33.129806 (+ 119us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.130363 (+ 557us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3747502}
I20250905 19:52:33.132158 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 302053e1611b4e22a9a2e7b60ae68f27 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:33.132536 2100 raft_consensus.cc:2747] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.132556 2568 raft_consensus.cc:491] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.132675 2568 raft_consensus.cc:513] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.132933 2379 tablet_replica.cc:331] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:33.133343 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 8d38452d3a0641ffa4af5d935f8ad9f0 already in progress: creating tablet
I20250905 19:52:33.135098 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:33.135576 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:33.136446 1171 tablet_bootstrap.cc:492] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.136778 1171 ts_tablet_manager.cc:1397] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.001s
I20250905 19:52:33.137804 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.138011 2568 leader_election.cc:290] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.137804 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.138522 1171 raft_consensus.cc:357] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.138947 1171 raft_consensus.cc:383] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.139089 1171 raft_consensus.cc:738] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:33.139125 848 leader_election.cc:343] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.139669 1171 consensus_queue.cc:260] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.140923 1171 ts_tablet_manager.cc:1428] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:33.141530 1171 tablet_bootstrap.cc:492] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.143497 2379 raft_consensus.cc:2241] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.144364 2379 raft_consensus.cc:2270] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.145946 1849 raft_consensus.cc:491] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.146056 1849 raft_consensus.cc:513] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.146472 2652 raft_consensus.cc:2270] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.146646 1849 leader_election.cc:290] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:33.123930 986 leader_election.cc:343] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.138599 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.146713 1170 tablet_bootstrap.cc:492] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.146860 1040 raft_consensus.cc:2466] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.145273 1169 tablet_bootstrap.cc:492] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.146991 1170 ts_tablet_manager.cc:1397] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.006s
W20250905 19:52:33.147101 984 leader_election.cc:343] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.147310 984 leader_election.cc:304] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.147399 2652 ts_tablet_manager.cc:1905] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.147936 2496 raft_consensus.cc:2747] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.147117 1169 ts_tablet_manager.cc:1397] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.033s user 0.001s sys 0.003s
I20250905 19:52:33.148694 1170 raft_consensus.cc:357] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.149031 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.149046 1170 raft_consensus.cc:383] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.149179 1170 raft_consensus.cc:738] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.149595 849 leader_election.cc:304] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.149597 1170 consensus_queue.cc:260] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.150588 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4dd2a96b8180438fbc722be1c18a0189" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.150777 1170 ts_tablet_manager.cc:1428] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:33.151041 712 leader_election.cc:304] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.151382 1170 tablet_bootstrap.cc:492] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.151409 1849 raft_consensus.cc:2747] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.151729 2577 raft_consensus.cc:2802] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.151804 2577 raft_consensus.cc:491] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.151865 2577 raft_consensus.cc:3058] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.154477 1170 tablet_bootstrap.cc:654] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.155565 2577 raft_consensus.cc:513] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.156312 2577 leader_election.cc:290] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.156446 2379 ts_tablet_manager.cc:1905] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.158361 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:33.158838 848 leader_election.cc:343] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.157218 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.161476 1040 raft_consensus.cc:3058] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:33.159891 986 consensus_peers.cc:489] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2683a36998674f739b41d2a57339954c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.164424 1040 raft_consensus.cc:2466] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.150664 1169 raft_consensus.cc:357] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.164979 849 leader_election.cc:304] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.167016 1170 tablet_bootstrap.cc:492] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.167352 1170 ts_tablet_manager.cc:1397] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.007s
I20250905 19:52:33.168035 1169 raft_consensus.cc:383] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.168113 1169 raft_consensus.cc:738] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.168407 1169 consensus_queue.cc:260] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.169169 1170 raft_consensus.cc:357] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.169512 1170 raft_consensus.cc:383] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.169647 1170 raft_consensus.cc:738] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.170059 1170 consensus_queue.cc:260] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.171301 1170 ts_tablet_manager.cc:1428] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20250905 19:52:33.171937 1170 tablet_bootstrap.cc:492] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.168968 1169 ts_tablet_manager.cc:1428] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.021s user 0.001s sys 0.001s
I20250905 19:52:33.175920 1170 tablet_bootstrap.cc:654] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.177691 2413 raft_consensus.cc:2802] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.178478 2413 raft_consensus.cc:695] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:33.178777 2413 consensus_queue.cc:237] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.179207 1169 tablet_bootstrap.cc:492] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.181536 768 raft_consensus.cc:3058] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.182796 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:33.183254 1169 tablet_bootstrap.cc:654] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.183400 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:33.165045 1171 tablet_bootstrap.cc:654] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.185446 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:33.185938 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:33.186214 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:33.188984 1170 tablet_bootstrap.cc:492] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.189320 1170 ts_tablet_manager.cc:1397] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.010s
I20250905 19:52:33.190940 775 raft_consensus.cc:1273] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Refusing update from remote peer 4eb3af79b01d4835bf6bf7b26bf016f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:52:33.191563 2685 consensus_queue.cc:1035] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:33.191692 1170 raft_consensus.cc:357] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.192140 1170 raft_consensus.cc:383] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.192296 1170 raft_consensus.cc:738] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.192797 1170 consensus_queue.cc:260] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.194015 1170 ts_tablet_manager.cc:1428] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:33.194643 1170 tablet_bootstrap.cc:492] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.198609 1170 tablet_bootstrap.cc:654] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.200313 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:33.200627 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:33.201426 2652 ts_tablet_manager.cc:1918] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.201519 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:33.202011 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:33.202647 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:33.203187 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:33.203640 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:33.205430 2652 log.cc:1199] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ba161282480e4736889559a2e37b195f
I20250905 19:52:33.208958 2376 ts_tablet_manager.cc:1918] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.209096 2376 log.cc:1199] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/54628965b82e470d811eff95ffc56c5f
I20250905 19:52:33.211476 2376 ts_tablet_manager.cc:1939] T 54628965b82e470d811eff95ffc56c5f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:33.212026 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 74) took 3867 ms. Trace:
I20250905 19:52:33.212122 2376 rpcz_store.cc:276] 0905 19:52:29.344449 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.344651 (+ 202us) service_pool.cc:225] Handling call
0905 19:52:33.121581 (+3776930us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:33.121593 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.178171 (+ 56578us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.208813 (+ 30642us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.212006 (+ 3193us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3772934}
I20250905 19:52:33.212481 2376 tablet_replica.cc:331] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:33.212690 2376 raft_consensus.cc:2241] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.212745 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 54628965b82e470d811eff95ffc56c5f (table default.Table_c815e30da2544766834b52eaefab63c4 [id=0b36fe8f4d004271aa9bbe78a44dc583]) successfully deleted
I20250905 19:52:33.213032 2376 raft_consensus.cc:2270] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.213596 2376 ts_tablet_manager.cc:1905] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.213594 1170 tablet_bootstrap.cc:492] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.214286 1170 ts_tablet_manager.cc:1397] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.020s user 0.010s sys 0.001s
I20250905 19:52:33.214385 1171 tablet_bootstrap.cc:492] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.214960 1171 ts_tablet_manager.cc:1397] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.074s user 0.004s sys 0.001s
I20250905 19:52:33.216583 2652 ts_tablet_manager.cc:1939] T ba161282480e4736889559a2e37b195f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.217199 1170 raft_consensus.cc:357] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.217602 1170 raft_consensus.cc:383] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.217732 1170 raft_consensus.cc:738] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.217851 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 65) took 1656 ms. Trace:
I20250905 19:52:33.218150 1170 consensus_queue.cc:260] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.218873 775 raft_consensus.cc:3058] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.219326 1170 ts_tablet_manager.cc:1428] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250905 19:52:33.222496 1171 raft_consensus.cc:357] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.222913 1171 raft_consensus.cc:383] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.223047 1171 raft_consensus.cc:738] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.223460 1171 consensus_queue.cc:260] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.222926 2652 rpcz_store.cc:276] 0905 19:52:31.561184 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.561241 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:33.122749 (+1561508us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:33.122758 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.177718 (+ 54960us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.201286 (+ 23568us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.217801 (+ 16515us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1561322}
I20250905 19:52:33.224709 1171 ts_tablet_manager.cc:1428] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20250905 19:52:33.224753 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ba161282480e4736889559a2e37b195f (table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]) successfully deleted
I20250905 19:52:33.225263 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 67) took 1654 ms. Trace:
I20250905 19:52:33.225327 1171 tablet_bootstrap.cc:492] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.226112 2652 rpcz_store.cc:276] 0905 19:52:31.570469 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571137 (+ 668us) service_pool.cc:225] Handling call
0905 19:52:33.225127 (+1653990us) ts_tablet_manager.cc:1145] Deleting tablet 6f3473785fff45d692bd50b1015ffe39
0905 19:52:33.225142 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.225246 (+ 104us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1652918}
W20250905 19:52:33.226688 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6f3473785fff45d692bd50b1015ffe39 because the tablet was not found. No further retry: Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:33.228277 2652 tablet_replica.cc:331] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.228564 2652 raft_consensus.cc:2241] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.228921 2652 raft_consensus.cc:2270] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.229102 2413 raft_consensus.cc:491] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.229233 2413 raft_consensus.cc:513] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.229590 2652 ts_tablet_manager.cc:1905] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.229736 1171 tablet_bootstrap.cc:654] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.230583 1170 tablet_bootstrap.cc:492] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.230684 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.231302 848 leader_election.cc:343] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.231626 2413 leader_election.cc:290] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.232409 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.232586 1040 raft_consensus.cc:2466] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.233417 849 leader_election.cc:304] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.233829 2738 raft_consensus.cc:2802] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.233934 2738 raft_consensus.cc:491] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.234027 2738 raft_consensus.cc:3058] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.237397 2738 raft_consensus.cc:513] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.238050 2738 leader_election.cc:290] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.238756 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.238924 1040 raft_consensus.cc:3058] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.241088 1542 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:33.241654 1040 raft_consensus.cc:2466] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.242367 849 leader_election.cc:304] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:33.243242 2413 raft_consensus.cc:2802] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.243418 2413 raft_consensus.cc:695] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:33.243775 2413 consensus_queue.cc:237] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.250795 2652 ts_tablet_manager.cc:1918] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.251802 2652 log.cc:1199] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:33.252555 2652 ts_tablet_manager.cc:1939] T 06850faa25de4b3d83f6bf98ccd370f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.253326 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 68) took 1682 ms. Trace:
I20250905 19:52:33.253793 2652 rpcz_store.cc:276] 0905 19:52:31.570592 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571161 (+ 569us) service_pool.cc:225] Handling call
0905 19:52:33.228228 (+1657067us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:33.228239 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.243094 (+ 14855us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.250673 (+ 7579us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.253309 (+ 2636us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1655876}
I20250905 19:52:33.254698 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 06850faa25de4b3d83f6bf98ccd370f0 (table default.Table_4ec0491dbe8841049cceaeee7c721211 [id=23b4c9dd0ca7459fa0d0a8b5f7033fd9]) successfully deleted
I20250905 19:52:33.255146 2652 tablet_replica.cc:331] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.255580 2652 raft_consensus.cc:2241] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.256158 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:33.256892 1170 tablet_bootstrap.cc:654] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:33.256901 848 leader_election.cc:343] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:33.258034 849 consensus_peers.cc:489] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8503f6918ffb44efa36214633c93e31d. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.258069 2379 ts_tablet_manager.cc:1918] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:33.258178 2379 log.cc:1199] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/3602094b96c14e66b997e8035fc53c40
I20250905 19:52:33.258458 2379 ts_tablet_manager.cc:1939] T 3602094b96c14e66b997e8035fc53c40 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:33.258759 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 72) took 3887 ms. Trace:
I20250905 19:52:33.258843 2379 rpcz_store.cc:276] 0905 19:52:29.370860 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371810 (+ 950us) service_pool.cc:225] Handling call
0905 19:52:33.132653 (+3760843us) ts_tablet_manager.cc:1145] Deleting tablet 3602094b96c14e66b997e8035fc53c40
0905 19:52:33.132663 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.160524 (+ 27861us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.257943 (+ 97419us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.258739 (+ 796us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":9492,"tablet-delete.queue_time_us":3750346}
I20250905 19:52:33.259182 2379 tablet_replica.cc:331] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:33.259347 2379 raft_consensus.cc:2241] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.259356 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 3602094b96c14e66b997e8035fc53c40 (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]) successfully deleted
I20250905 19:52:33.259505 2379 raft_consensus.cc:2270] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.260085 2379 ts_tablet_manager.cc:1905] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.267831 2413 consensus_queue.cc:1035] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:33.259644 1169 tablet_bootstrap.cc:492] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.290032 1169 ts_tablet_manager.cc:1397] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.111s user 0.002s sys 0.005s
I20250905 19:52:33.291884 1169 raft_consensus.cc:357] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.292199 1169 raft_consensus.cc:383] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.292304 1169 raft_consensus.cc:738] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.292622 1169 consensus_queue.cc:260] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.293322 1169 ts_tablet_manager.cc:1428] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:33.293678 1169 tablet_bootstrap.cc:492] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.295145 1169 tablet_bootstrap.cc:654] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.289731 2652 raft_consensus.cc:2270] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.295913 1171 tablet_bootstrap.cc:492] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.296447 2652 ts_tablet_manager.cc:1905] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.301853 1171 ts_tablet_manager.cc:1397] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.077s user 0.010s sys 0.016s
I20250905 19:52:33.303874 1171 raft_consensus.cc:357] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.304384 1171 raft_consensus.cc:383] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.304757 1171 raft_consensus.cc:738] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:33.313709 848 consensus_peers.cc:489] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.314172 1169 tablet_bootstrap.cc:492] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.316102 1169 ts_tablet_manager.cc:1397] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.002s
I20250905 19:52:33.315254 1171 consensus_queue.cc:260] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.316907 1169 raft_consensus.cc:357] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.317250 2785 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:33.318013 1169 raft_consensus.cc:383] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.318167 1169 raft_consensus.cc:738] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.318167 1171 ts_tablet_manager.cc:1428] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.016s user 0.001s sys 0.003s
I20250905 19:52:33.318454 1169 consensus_queue.cc:260] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.318969 1169 ts_tablet_manager.cc:1428] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:33.319275 1169 tablet_bootstrap.cc:492] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.319216 1171 tablet_bootstrap.cc:492] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.320770 1169 tablet_bootstrap.cc:654] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:33.325914 849 consensus_peers.cc:489] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: f48f9f789b6e46ec871a0fd007770dfc. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.326658 1170 tablet_bootstrap.cc:492] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.327057 1170 ts_tablet_manager.cc:1397] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.004s
I20250905 19:52:33.328253 2785 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:33.328797 2652 ts_tablet_manager.cc:1918] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.328954 2652 log.cc:1199] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/5b0d4be788f841c2860f5dd65275b623
I20250905 19:52:33.328990 1170 raft_consensus.cc:357] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.329334 2652 ts_tablet_manager.cc:1939] T 5b0d4be788f841c2860f5dd65275b623 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.329416 1170 raft_consensus.cc:383] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250905 19:52:33.330850 849 consensus_peers.cc:489] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c46ea498d6a14d1ebdffea415cfc7114. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.331797 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 70) took 1761 ms. Trace:
I20250905 19:52:33.334427 775 raft_consensus.cc:1273] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Refusing update from remote peer 4eb3af79b01d4835bf6bf7b26bf016f5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:52:33.335031 2496 consensus_queue.cc:1035] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:33.335053 2652 rpcz_store.cc:276] 0905 19:52:31.570816 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571201 (+ 385us) service_pool.cc:225] Handling call
0905 19:52:33.255092 (+1683891us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:33.255102 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.304157 (+ 49055us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.328687 (+ 24530us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.331775 (+ 3088us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1682586}
I20250905 19:52:33.335556 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 5b0d4be788f841c2860f5dd65275b623 (table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]) successfully deleted
I20250905 19:52:33.339741 1169 tablet_bootstrap.cc:492] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.339923 1169 ts_tablet_manager.cc:1397] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.000s
I20250905 19:52:33.340677 1169 raft_consensus.cc:357] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.340999 1169 raft_consensus.cc:383] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.341097 1169 raft_consensus.cc:738] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.341360 1169 consensus_queue.cc:260] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.341912 1169 ts_tablet_manager.cc:1428] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
W20250905 19:52:33.343130 848 consensus_peers.cc:489] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.346236 1571 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 651 } } ignore_live_leader: false dest_uuid: "e38333af29db4bf6af41ee5a25d6701e" is_pre_election: true
W20250905 19:52:33.346720 986 consensus_peers.cc:489] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8edf4b54974c427a855eee5792d75292. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.347530 1171 tablet_bootstrap.cc:654] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.350646 2738 raft_consensus.cc:491] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.350791 2738 raft_consensus.cc:513] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.351300 2738 leader_election.cc:290] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.356154 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.358719 848 leader_election.cc:343] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.329619 1170 raft_consensus.cc:738] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.359840 1170 consensus_queue.cc:260] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.361019 769 raft_consensus.cc:3058] T 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.361266 1170 ts_tablet_manager.cc:1428] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.034s user 0.003s sys 0.000s
I20250905 19:52:33.366606 2785 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:33.367726 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.367935 1040 raft_consensus.cc:2466] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.369865 2376 ts_tablet_manager.cc:1918] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.370129 2376 log.cc:1199] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a3805127dd78489e8545da7414ead904
I20250905 19:52:33.370221 1171 tablet_bootstrap.cc:492] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.370347 1171 ts_tablet_manager.cc:1397] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.052s user 0.006s sys 0.000s
I20250905 19:52:33.370371 849 leader_election.cc:304] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.370632 2738 raft_consensus.cc:2802] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.370704 2738 raft_consensus.cc:491] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.370766 2738 raft_consensus.cc:3058] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.371088 2376 ts_tablet_manager.cc:1939] T a3805127dd78489e8545da7414ead904 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:33.371250 1171 raft_consensus.cc:357] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.371409 1171 raft_consensus.cc:383] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.371465 1171 raft_consensus.cc:738] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.371508 2568 consensus_queue.cc:1035] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:33.371651 1171 consensus_queue.cc:260] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.372184 1171 ts_tablet_manager.cc:1428] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:33.372444 1171 tablet_bootstrap.cc:492] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.374365 1171 tablet_bootstrap.cc:654] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.374783 775 raft_consensus.cc:3058] T 31e332cdfedd4ff6b5026baa73e2caf3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.384641 1170 tablet_bootstrap.cc:492] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.386072 2738 raft_consensus.cc:513] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.386390 2577 raft_consensus.cc:491] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.386538 2577 raft_consensus.cc:513] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.387132 2577 leader_election.cc:290] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.390456 2738 leader_election.cc:290] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.396862 2684 raft_consensus.cc:491] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.397019 2684 raft_consensus.cc:513] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.397663 2684 leader_election.cc:290] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.398201 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:33.398821 986 leader_election.cc:343] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.404230 1169 tablet_bootstrap.cc:492] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.406078 1169 tablet_bootstrap.cc:654] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.409569 2738 raft_consensus.cc:491] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.409799 2738 raft_consensus.cc:513] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.411203 2738 leader_election.cc:290] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.412997 2652 tablet_replica.cc:331] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.413290 2652 raft_consensus.cc:2241] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.417106 2652 raft_consensus.cc:2270] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.418939 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:33.420677 1171 tablet_bootstrap.cc:492] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.420855 1171 ts_tablet_manager.cc:1397] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.009s
I20250905 19:52:33.420732 2652 ts_tablet_manager.cc:1905] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.421677 1171 raft_consensus.cc:357] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.421861 1171 raft_consensus.cc:383] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.421927 1171 raft_consensus.cc:738] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.422122 1171 consensus_queue.cc:260] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.422628 1171 ts_tablet_manager.cc:1428] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:33.422919 1171 tablet_bootstrap.cc:492] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:33.428215 1479 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet e10b4e82c4bc429b9a42db39c03a5267 consensus state has no leader
W20250905 19:52:33.428189 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 57051.6us
I20250905 19:52:33.406795 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 72) took 4091 ms. Trace:
I20250905 19:52:33.430410 2376 rpcz_store.cc:276] 0905 19:52:29.315453 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.315683 (+ 230us) service_pool.cc:225] Handling call
0905 19:52:33.212434 (+3896751us) ts_tablet_manager.cc:1145] Deleting tablet a3805127dd78489e8545da7414ead904
0905 19:52:33.212444 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.282209 (+ 69765us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.369595 (+ 87386us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.406520 (+ 36925us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3861986}
I20250905 19:52:33.390339 1170 tablet_bootstrap.cc:654] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.431529 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a3805127dd78489e8545da7414ead904 (table default.Table_e65ca281d8804277a83ab7bfc2ab9cca [id=3ed6b530762e4290aa5c45ee1a55c100]) successfully deleted
I20250905 19:52:33.391469 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:33.391556 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0c478efe044ad0867e47f49374a66b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.397603 2379 ts_tablet_manager.cc:1918] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.404781 2160 raft_consensus.cc:491] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.432593 2379 log.cc:1199] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:33.398092 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.433131 984 leader_election.cc:343] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:33.433187 848 leader_election.cc:343] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.433247 984 leader_election.cc:304] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:33.433337 848 leader_election.cc:343] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.432628 2160 raft_consensus.cc:513] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.433652 2379 ts_tablet_manager.cc:1939] T ac5c88cf8d15433a940e937f0a225582 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:33.434254 2160 leader_election.cc:290] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.411023 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.434504 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 84) took 4057 ms. Trace:
I20250905 19:52:33.434793 2684 raft_consensus.cc:2747] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.434666 2379 rpcz_store.cc:276] 0905 19:52:29.376583 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.378075 (+ 1492us) service_pool.cc:225] Handling call
0905 19:52:33.259136 (+3881061us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:33.259146 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.263202 (+ 4056us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.397281 (+134079us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.434439 (+ 37158us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3876821}
I20250905 19:52:33.435500 2379 tablet_replica.cc:331] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:33.435925 2379 raft_consensus.cc:2241] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:33.436074 848 leader_election.cc:343] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:33.435921 711 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:33.436349 2379 raft_consensus.cc:2270] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.438500 2379 ts_tablet_manager.cc:1905] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.439309 2684 raft_consensus.cc:491] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.439427 2684 raft_consensus.cc:513] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.440142 2684 leader_election.cc:290] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.441118 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c824f5591044c4198a51a25661f71e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.441414 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0c478efe044ad0867e47f49374a66b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.441584 1044 raft_consensus.cc:2466] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:33.441603 984 leader_election.cc:343] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.442489 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.442670 1044 raft_consensus.cc:3058] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.445477 1044 raft_consensus.cc:2466] T db098a883a61474fb06cb166041ee3e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.447414 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.447616 1044 raft_consensus.cc:2466] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.449316 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c824f5591044c4198a51a25661f71e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.449506 909 raft_consensus.cc:2466] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:33.449540 1171 tablet_bootstrap.cc:654] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.449920 986 leader_election.cc:304] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.450371 2684 raft_consensus.cc:2802] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.450460 2684 raft_consensus.cc:491] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.450536 2684 raft_consensus.cc:3058] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.451254 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:33.452909 2684 raft_consensus.cc:513] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.453924 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c824f5591044c4198a51a25661f71e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:33.454447 984 leader_election.cc:343] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.455354 2684 leader_election.cc:290] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.455243 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c824f5591044c4198a51a25661f71e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:33.455911 909 raft_consensus.cc:3058] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.458707 909 raft_consensus.cc:2466] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:33.459180 986 leader_election.cc:304] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.460017 2684 raft_consensus.cc:2802] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.460276 2684 raft_consensus.cc:695] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:33.460589 2684 consensus_queue.cc:237] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.461093 849 leader_election.cc:304] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.461478 849 leader_election.cc:304] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.462533 2376 tablet_replica.cc:331] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:33.462956 2376 raft_consensus.cc:2241] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.464551 2160 raft_consensus.cc:491] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.464699 2160 raft_consensus.cc:513] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.465291 2160 leader_election.cc:290] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.466111 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.467931 2738 raft_consensus.cc:2802] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.468135 2738 raft_consensus.cc:491] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.468246 2738 raft_consensus.cc:3058] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.468742 2160 raft_consensus.cc:491] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.468865 2160 raft_consensus.cc:513] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.469424 2160 leader_election.cc:290] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.471431 2738 raft_consensus.cc:513] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.472579 2738 leader_election.cc:290] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.472769 2738 raft_consensus.cc:2802] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.473011 2738 raft_consensus.cc:695] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
W20250905 19:52:33.472311 711 leader_election.cc:343] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:33.474294 2376 raft_consensus.cc:2270] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.475265 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:33.475607 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0c478efe044ad0867e47f49374a66b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:33.475688 2376 ts_tablet_manager.cc:1905] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:33.476456 848 leader_election.cc:343] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.478710 774 raft_consensus.cc:3058] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.484489 1171 tablet_bootstrap.cc:492] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.484658 1171 ts_tablet_manager.cc:1397] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.062s user 0.008s sys 0.005s
I20250905 19:52:33.485499 1171 raft_consensus.cc:357] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.485670 1171 raft_consensus.cc:383] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.485730 1171 raft_consensus.cc:738] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.485936 1171 consensus_queue.cc:260] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.486430 1171 ts_tablet_manager.cc:1428] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:33.486711 1171 tablet_bootstrap.cc:492] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.487890 2738 consensus_queue.cc:237] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.488948 1170 tablet_bootstrap.cc:492] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.489300 1170 ts_tablet_manager.cc:1397] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.128s user 0.006s sys 0.001s
W20250905 19:52:33.488376 849 consensus_peers.cc:489] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 44769f1170354870ae2d34fdb67ffbae. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.490694 849 leader_election.cc:304] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.491128 1170 raft_consensus.cc:357] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.491554 1170 raft_consensus.cc:383] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.491750 1170 raft_consensus.cc:738] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.492290 1170 consensus_queue.cc:260] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.492537 2789 raft_consensus.cc:2802] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.492666 2789 raft_consensus.cc:491] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.492786 2789 raft_consensus.cc:3058] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.493772 1170 ts_tablet_manager.cc:1428] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20250905 19:52:33.488517 1171 tablet_bootstrap.cc:654] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.494424 1170 tablet_bootstrap.cc:492] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.509958 1171 tablet_bootstrap.cc:492] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.510113 1171 ts_tablet_manager.cc:1397] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.023s user 0.003s sys 0.004s
I20250905 19:52:33.511590 1171 raft_consensus.cc:357] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.511787 1171 raft_consensus.cc:383] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.511839 1171 raft_consensus.cc:738] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.512063 1171 consensus_queue.cc:260] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.512575 1171 ts_tablet_manager.cc:1428] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:33.513051 1171 tablet_bootstrap.cc:492] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.513154 2685 raft_consensus.cc:491] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.513302 2685 raft_consensus.cc:513] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.513896 2685 leader_election.cc:290] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.514889 1171 tablet_bootstrap.cc:654] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.514860 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.515794 984 leader_election.cc:343] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.516098 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:33.516587 986 leader_election.cc:343] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.516703 986 leader_election.cc:304] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.517323 2685 raft_consensus.cc:2747] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.522429 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:33.522830 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:33.524734 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ac5c88cf8d15433a940e937f0a225582 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:33.525277 1171 tablet_bootstrap.cc:492] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.525414 1171 ts_tablet_manager.cc:1397] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.003s
I20250905 19:52:33.526206 1171 raft_consensus.cc:357] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.526386 1171 raft_consensus.cc:383] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.526441 1171 raft_consensus.cc:738] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.526613 1171 consensus_queue.cc:260] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.527076 1171 ts_tablet_manager.cc:1428] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:33.527331 1171 tablet_bootstrap.cc:492] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.528007 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:33.529428 1171 tablet_bootstrap.cc:654] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.529609 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:33.531147 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:33.531911 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:33.533215 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:33.541013 1171 tablet_bootstrap.cc:492] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.541160 1171 ts_tablet_manager.cc:1397] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.003s
I20250905 19:52:33.541958 1171 raft_consensus.cc:357] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.542124 1171 raft_consensus.cc:383] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.542178 1171 raft_consensus.cc:738] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.542382 1171 consensus_queue.cc:260] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.543308 1171 ts_tablet_manager.cc:1428] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:33.544348 1171 tablet_bootstrap.cc:492] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.546429 1169 tablet_bootstrap.cc:492] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.546510 1171 tablet_bootstrap.cc:654] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.550781 2789 raft_consensus.cc:513] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.556773 1169 ts_tablet_manager.cc:1397] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.153s user 0.011s sys 0.001s
I20250905 19:52:33.557672 1169 raft_consensus.cc:357] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.557902 1169 raft_consensus.cc:383] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.557993 1169 raft_consensus.cc:738] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.558295 1169 consensus_queue.cc:260] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.567831 2789 leader_election.cc:290] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:33.553622 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 48100.5us
I20250905 19:52:33.569167 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0fb8fe93061848a0b055979bad10f08b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.559249 1171 tablet_bootstrap.cc:492] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.569746 1171 ts_tablet_manager.cc:1397] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.004s
I20250905 19:52:33.570111 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:33.570595 1171 raft_consensus.cc:357] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.570789 1171 raft_consensus.cc:383] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.570843 1171 raft_consensus.cc:738] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:33.570859 848 leader_election.cc:343] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.571022 1171 consensus_queue.cc:260] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.571542 1171 ts_tablet_manager.cc:1428] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:33.571830 1171 tablet_bootstrap.cc:492] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.573752 1171 tablet_bootstrap.cc:654] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.575551 2376 ts_tablet_manager.cc:1918] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.575688 2376 log.cc:1199] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/3de936d2309a411eabbd59f7b73ce7f3
I20250905 19:52:33.577176 1170 tablet_bootstrap.cc:654] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.583271 2160 raft_consensus.cc:491] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.583410 2160 raft_consensus.cc:513] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.584031 2160 leader_election.cc:290] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:33.585242 712 leader_election.cc:343] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:33.585379 712 leader_election.cc:304] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.586485 2160 raft_consensus.cc:2747] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.589558 1171 tablet_bootstrap.cc:492] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.589711 1171 ts_tablet_manager.cc:1397] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.002s
I20250905 19:52:33.592845 2811 tablet.cc:1722] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.596324 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:33.597335 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:33.602564 1170 tablet_bootstrap.cc:492] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.602773 1170 ts_tablet_manager.cc:1397] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.109s user 0.004s sys 0.000s
I20250905 19:52:33.602777 1171 raft_consensus.cc:357] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.602952 1171 raft_consensus.cc:383] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.603020 1171 raft_consensus.cc:738] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.603245 1171 consensus_queue.cc:260] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.603777 1171 ts_tablet_manager.cc:1428] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250905 19:52:33.604100 1171 tablet_bootstrap.cc:492] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.604291 1170 raft_consensus.cc:357] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.604705 712 leader_election.cc:304] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.605178 2160 raft_consensus.cc:2747] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:33.605743 712 leader_election.cc:343] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:33.605871 712 leader_election.cc:304] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.606225 2160 raft_consensus.cc:2747] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.607509 1171 tablet_bootstrap.cc:654] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.611281 2376 ts_tablet_manager.cc:1939] T 3de936d2309a411eabbd59f7b73ce7f3 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:33.604727 1170 raft_consensus.cc:383] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.611902 1170 raft_consensus.cc:738] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.612496 1170 consensus_queue.cc:260] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.622854 1170 ts_tablet_manager.cc:1428] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.020s user 0.002s sys 0.001s
I20250905 19:52:33.625658 2379 ts_tablet_manager.cc:1918] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:33.626106 2379 log.cc:1199] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/68f2ccd8a7a64b89b44f2d6767ec3466
I20250905 19:52:33.627125 2379 ts_tablet_manager.cc:1939] T 68f2ccd8a7a64b89b44f2d6767ec3466 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:33.627916 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 74) took 4256 ms. Trace:
I20250905 19:52:33.629133 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 75) took 4280 ms. Trace:
I20250905 19:52:33.639981 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:33.640939 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:33.641520 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:33.641791 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]). Expected version 1 got 0
I20250905 19:52:33.642623 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:33.642848 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:33.643086 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:33.646833 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239539ac19124de192cadae494720a2b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.629348 2376 rpcz_store.cc:276] 0905 19:52:29.348880 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.348962 (+ 82us) service_pool.cc:225] Handling call
0905 19:52:33.431178 (+4082216us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:33.462403 (+ 31225us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.572096 (+109693us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.575402 (+ 3306us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.629060 (+ 53658us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":10210,"tablet-delete.queue_time_us":4072630}
I20250905 19:52:33.648984 2376 tablet_replica.cc:331] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:33.649121 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 3de936d2309a411eabbd59f7b73ce7f3 (table default.Table_88ff02c5dce2458784f09b64cee688f1 [id=378b42fcb4e741e4b4755deb4b0ef51a]) successfully deleted
I20250905 19:52:33.649209 2376 raft_consensus.cc:2241] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.649523 2376 raft_consensus.cc:2270] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.650070 2376 ts_tablet_manager.cc:1905] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.653260 2820 tablet.cc:1722] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.654460 1171 tablet_bootstrap.cc:492] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.654593 1171 ts_tablet_manager.cc:1397] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.051s user 0.014s sys 0.008s
I20250905 19:52:33.628243 2379 rpcz_store.cc:276] 0905 19:52:29.371011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371831 (+ 820us) service_pool.cc:225] Handling call
0905 19:52:33.435367 (+4063536us) ts_tablet_manager.cc:1145] Deleting tablet 68f2ccd8a7a64b89b44f2d6767ec3466
0905 19:52:33.435415 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.516092 (+ 80677us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.579854 (+ 63762us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.627856 (+ 48002us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4052885}
I20250905 19:52:33.623692 1170 tablet_bootstrap.cc:492] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.657882 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 68f2ccd8a7a64b89b44f2d6767ec3466 (table default.Table_be613708cf304dc2856d98a4e1f6920f [id=15dd8b7f1d00449a8f3f25ee97763002]) successfully deleted
I20250905 19:52:33.659348 2379 tablet_replica.cc:331] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:33.660077 2826 tablet.cc:1722] T efb70327f55844879f89449a7116010f P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.665053 2812 tablet.cc:1722] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:33.559948 1169 ts_tablet_manager.cc:1428] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:33.569605 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8e0c478efe044ad0867e47f49374a66b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.583001 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:33.666124 1036 raft_consensus.cc:3058] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.584602 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "239539ac19124de192cadae494720a2b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.619534 2496 raft_consensus.cc:491] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.668318 2496 raft_consensus.cc:513] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.669003 2496 leader_election.cc:290] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.669714 1037 raft_consensus.cc:3058] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.672858 1036 raft_consensus.cc:2466] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.673897 1037 raft_consensus.cc:2466] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:33.549674 2652 ts_tablet_manager.cc:1918] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.674324 2652 log.cc:1199] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/53ac46ce05694bc2b53e195638be9e23
I20250905 19:52:33.659622 2824 tablet.cc:1722] T ea6df8634a314ceabcf3fd004bf7e241 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:33.674721 2652 ts_tablet_manager.cc:1939] T 53ac46ce05694bc2b53e195638be9e23 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.675509 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 71) took 2104 ms. Trace:
I20250905 19:52:33.643918 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
W20250905 19:52:33.680790 849 consensus_peers.cc:489] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: cbfd03ad0bd34b6ca86e7b07a62b2a8e. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.680213 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:33.685256 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:33.668732 1170 tablet_bootstrap.cc:654] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.669626 2379 raft_consensus.cc:2241] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.692313 2379 raft_consensus.cc:2270] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.670222 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.694834 2379 ts_tablet_manager.cc:1905] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.700409 2652 rpcz_store.cc:276] 0905 19:52:31.570909 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571222 (+ 313us) service_pool.cc:225] Handling call
0905 19:52:33.336187 (+1764965us) ts_tablet_manager.cc:1145] Deleting tablet 53ac46ce05694bc2b53e195638be9e23
0905 19:52:33.367620 (+ 31433us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.451610 (+ 83990us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.549564 (+ 97954us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.675492 (+125928us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1763553}
I20250905 19:52:33.701403 2652 tablet_replica.cc:331] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:33.701640 2652 raft_consensus.cc:2241] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.701920 2652 raft_consensus.cc:2270] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.702684 2652 ts_tablet_manager.cc:1905] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.703388 2496 raft_consensus.cc:491] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.703516 2496 raft_consensus.cc:513] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.704241 2496 leader_election.cc:290] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.680029 1169 tablet_bootstrap.cc:492] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:33.693065 984 leader_election.cc:343] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.707190 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.707254 1171 raft_consensus.cc:357] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.707474 1171 raft_consensus.cc:383] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.707513 2413 raft_consensus.cc:491] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.707536 1171 raft_consensus.cc:738] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.707639 2413 raft_consensus.cc:513] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.707775 1171 consensus_queue.cc:260] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.701190 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 53ac46ce05694bc2b53e195638be9e23 (table default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]) successfully deleted
I20250905 19:52:33.709206 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d949c3c24e40fe8fefd3a5758d57ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.709760 1169 tablet_bootstrap.cc:654] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:33.709785 848 leader_election.cc:343] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.710579 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d949c3c24e40fe8fefd3a5758d57ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.691797 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
W20250905 19:52:33.711304 984 leader_election.cc:343] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.711373 1171 ts_tablet_manager.cc:1428] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.057s user 0.003s sys 0.000s
I20250905 19:52:33.711699 1171 tablet_bootstrap.cc:492] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.714371 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
W20250905 19:52:33.720607 986 leader_election.cc:343] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.720785 986 leader_election.cc:304] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.720979 2811 tablet.cc:1722] T efb70327f55844879f89449a7116010f P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.721175 2496 raft_consensus.cc:2747] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.721496 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d949c3c24e40fe8fefd3a5758d57ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.721702 906 raft_consensus.cc:2466] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:33.722219 2413 leader_election.cc:290] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.722945 986 leader_election.cc:304] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.723291 2496 raft_consensus.cc:2802] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.723358 2496 raft_consensus.cc:491] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.723412 2496 raft_consensus.cc:3058] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.722635 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:33.732434 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:33.735091 2496 raft_consensus.cc:513] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.736231 2496 leader_election.cc:290] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.736990 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d949c3c24e40fe8fefd3a5758d57ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:33.737169 914 raft_consensus.cc:3058] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.740386 914 raft_consensus.cc:2466] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:33.740978 986 leader_election.cc:304] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:33.741397 2496 raft_consensus.cc:2802] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.741549 2496 raft_consensus.cc:695] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:33.741942 2496 consensus_queue.cc:237] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.742205 2379 ts_tablet_manager.cc:1918] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.742478 2379 log.cc:1199] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:33.743263 2379 ts_tablet_manager.cc:1939] T b0702063ec1c4f6fba58e6d4f3912212 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:33.748363 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d949c3c24e40fe8fefd3a5758d57ea" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:33.749194 984 leader_election.cc:343] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.750790 1170 tablet_bootstrap.cc:492] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.753073 1170 ts_tablet_manager.cc:1397] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.130s user 0.009s sys 0.001s
I20250905 19:52:33.756640 1169 tablet_bootstrap.cc:492] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.758778 1169 ts_tablet_manager.cc:1397] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.079s user 0.005s sys 0.001s
I20250905 19:52:33.758855 2823 tablet.cc:1722] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:33.760330 1169 raft_consensus.cc:357] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:33.760871 848 consensus_peers.cc:489] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:33.761718 848 consensus_peers.cc:489] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.761970 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:33.764487 1169 raft_consensus.cc:383] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.764664 1169 raft_consensus.cc:738] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.768863 2819 tablet.cc:1722] T ea6df8634a314ceabcf3fd004bf7e241 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
W20250905 19:52:33.771497 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 90501us
I20250905 19:52:33.771633 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 71) took 4400 ms. Trace:
I20250905 19:52:33.771782 849 leader_election.cc:304] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.771903 2379 rpcz_store.cc:276] 0905 19:52:29.370780 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.371780 (+ 1000us) service_pool.cc:225] Handling call
0905 19:52:33.657662 (+4285882us) ts_tablet_manager.cc:1145] Deleting tablet b0702063ec1c4f6fba58e6d4f3912212
0905 19:52:33.658716 (+ 1054us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.735043 (+ 76327us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.741989 (+ 6946us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.771566 (+ 29577us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4275089}
I20250905 19:52:33.772558 849 leader_election.cc:304] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.772806 2379 tablet_replica.cc:331] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:33.773145 2738 raft_consensus.cc:2802] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.773232 2422 raft_consensus.cc:2802] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:33.773321 2738 raft_consensus.cc:695] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:33.773286 2379 raft_consensus.cc:2241] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.773352 2422 raft_consensus.cc:695] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:33.774282 2738 consensus_queue.cc:237] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.775171 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet b0702063ec1c4f6fba58e6d4f3912212 (table default.Table_46df29d8bee2462fb784be95829c6ad0 [id=3ed507a38d1049fa893efa7be65f3609]) successfully deleted
I20250905 19:52:33.764943 1169 consensus_queue.cc:260] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.776494 1169 ts_tablet_manager.cc:1428] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250905 19:52:33.777091 1169 tablet_bootstrap.cc:492] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.774286 2422 consensus_queue.cc:237] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.782946 1171 tablet_bootstrap.cc:654] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.783144 1169 tablet_bootstrap.cc:654] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.780736 2832 tablet.cc:1722] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.774437 2379 raft_consensus.cc:2270] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.782064 1170 raft_consensus.cc:357] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.790769 1170 raft_consensus.cc:383] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.792695 1170 raft_consensus.cc:738] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.795312 2812 tablet.cc:1722] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.801363 1169 tablet_bootstrap.cc:492] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:33.801766 1169 ts_tablet_manager.cc:1397] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.025s user 0.011s sys 0.001s
I20250905 19:52:33.805315 712 leader_election.cc:304] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.806027 1849 raft_consensus.cc:2747] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.806717 2379 ts_tablet_manager.cc:1905] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.806788 1170 consensus_queue.cc:260] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.809904 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:33.814301 1170 ts_tablet_manager.cc:1428] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.060s user 0.002s sys 0.002s
I20250905 19:52:33.814155 1547 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:33.819070 1170 tablet_bootstrap.cc:492] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.818928 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:33.827327 2685 raft_consensus.cc:491] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.827473 2685 raft_consensus.cc:513] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.828104 2685 leader_election.cc:290] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.828855 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95b32fff7d4745c79185a5d894ef3342" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.830111 984 leader_election.cc:343] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.832099 2824 tablet.cc:1722] T 239539ac19124de192cadae494720a2b P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.835480 2811 tablet.cc:1722] T 239539ac19124de192cadae494720a2b P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.836742 776 raft_consensus.cc:3058] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.840639 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97d949c3c24e40fe8fefd3a5758d57ea" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.843923 1849 raft_consensus.cc:491] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.844133 1849 raft_consensus.cc:513] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.845028 1849 leader_election.cc:290] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.846002 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.854693 1170 tablet_bootstrap.cc:654] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.866883 1169 raft_consensus.cc:357] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.867430 1169 raft_consensus.cc:383] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.867570 1169 raft_consensus.cc:738] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.868023 1169 consensus_queue.cc:260] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.869232 1169 ts_tablet_manager.cc:1428] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.067s user 0.000s sys 0.002s
I20250905 19:52:33.869800 1169 tablet_bootstrap.cc:492] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:33.870280 1171 tablet_bootstrap.cc:492] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.870432 1171 ts_tablet_manager.cc:1397] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.159s user 0.008s sys 0.019s
I20250905 19:52:33.871264 1171 raft_consensus.cc:357] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.871446 1171 raft_consensus.cc:383] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.871510 1171 raft_consensus.cc:738] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.871733 1171 consensus_queue.cc:260] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.872407 1171 ts_tablet_manager.cc:1428] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:33.872723 1171 tablet_bootstrap.cc:492] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.876817 1171 tablet_bootstrap.cc:654] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.881167 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:33.881654 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:33.882228 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:33.882829 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:33.884723 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:33.885542 2838 raft_consensus.cc:491] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.885679 2838 raft_consensus.cc:513] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.886322 2838 leader_election.cc:290] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.886369 1169 tablet_bootstrap.cc:654] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.887064 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95b32fff7d4745c79185a5d894ef3342" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.887346 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d095804b51b4205a23ac3d3106312f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.887898 986 leader_election.cc:343] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:33.888036 848 consensus_peers.cc:489] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.888036 986 leader_election.cc:304] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:33.888218 848 leader_election.cc:343] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.888481 2684 raft_consensus.cc:2747] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.889331 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:33.890218 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:33.891571 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:33.892968 2838 raft_consensus.cc:491] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.893085 1171 tablet_bootstrap.cc:492] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:33.893119 2838 raft_consensus.cc:513] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.893224 1171 ts_tablet_manager.cc:1397] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.005s
I20250905 19:52:33.893718 2838 leader_election.cc:290] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.894048 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c14dc571b586405f8fd1eaa452998978" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.894083 1171 raft_consensus.cc:357] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.894291 1171 raft_consensus.cc:383] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.894356 1171 raft_consensus.cc:738] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:33.894579 848 leader_election.cc:343] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.894646 1171 consensus_queue.cc:260] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.895411 769 raft_consensus.cc:3058] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.895905 1171 ts_tablet_manager.cc:1428] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:33.896275 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:33.896628 1540 catalog_manager.cc:5582] T f2d929ec75d34cddad5286d8e37fcefb P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:33.897364 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:33.911511 2652 ts_tablet_manager.cc:1918] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.911648 2652 log.cc:1199] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/091ef981d6b54756830927ea111e78aa
I20250905 19:52:33.911998 2652 ts_tablet_manager.cc:1939] T 091ef981d6b54756830927ea111e78aa P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:33.913959 2379 ts_tablet_manager.cc:1918] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.914080 2379 log.cc:1199] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:33.914132 2577 consensus_queue.cc:1035] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:33.914439 2379 ts_tablet_manager.cc:1939] T a5f076d9917f40d899d7422ad974fe0f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:33.915310 848 consensus_peers.cc:489] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.915584 849 leader_election.cc:304] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.916124 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0e05db608a154b1ca0f9a1ed06c7fbd3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:33.916623 848 consensus_peers.cc:489] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:33.916785 2838 raft_consensus.cc:2747] T 97d949c3c24e40fe8fefd3a5758d57ea P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.918915 2376 ts_tablet_manager.cc:1918] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:33.919036 2376 log.cc:1199] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/091ef981d6b54756830927ea111e78aa
I20250905 19:52:33.919677 2376 ts_tablet_manager.cc:1939] T 091ef981d6b54756830927ea111e78aa P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:33.921984 1171 tablet_bootstrap.cc:492] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:33.924588 712 leader_election.cc:304] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:33.925060 1849 raft_consensus.cc:2747] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.925277 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:33.925704 1170 tablet_bootstrap.cc:492] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:33.925848 1170 ts_tablet_manager.cc:1397] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.004s
I20250905 19:52:33.925860 1171 tablet_bootstrap.cc:654] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.926654 1170 raft_consensus.cc:357] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.926745 2838 raft_consensus.cc:491] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.926836 1170 raft_consensus.cc:383] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:33.926898 1170 raft_consensus.cc:738] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:33.926870 2838 raft_consensus.cc:513] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.927516 2838 leader_election.cc:290] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.927413 1170 consensus_queue.cc:260] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.928575 1170 ts_tablet_manager.cc:1428] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250905 19:52:33.929337 2838 raft_consensus.cc:491] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.929490 2838 raft_consensus.cc:513] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.930120 2838 leader_election.cc:290] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.930765 1849 raft_consensus.cc:491] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.930900 1849 raft_consensus.cc:513] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.931455 1849 leader_election.cc:290] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:33.931779 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.932122 2838 raft_consensus.cc:491] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.932247 2838 raft_consensus.cc:513] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.932591 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:33.932358 1170 tablet_bootstrap.cc:492] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:33.932794 2838 leader_election.cc:290] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:33.932762 848 leader_election.cc:343] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.932955 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a2ff206366ad4344b4df5fc6a5584f8a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.933282 1540 catalog_manager.cc:5582] T 4d6f2e44855d4312b1c0367145e3cee8 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:33.933523 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:33.934746 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
W20250905 19:52:33.935112 848 leader_election.cc:343] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:33.935369 848 leader_election.cc:343] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.935755 775 raft_consensus.cc:3058] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.939785 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:33.943203 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:33.944888 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:33.945451 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d095804b51b4205a23ac3d3106312f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.945677 1050 raft_consensus.cc:2466] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.945871 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c14dc571b586405f8fd1eaa452998978" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.946063 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
I20250905 19:52:33.946039 1035 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.946291 1035 raft_consensus.cc:2466] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.947135 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:33.946024 1045 raft_consensus.cc:2466] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.950065 2684 raft_consensus.cc:491] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.950220 2684 raft_consensus.cc:513] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.950974 2684 leader_election.cc:290] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.951726 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.952360 984 leader_election.cc:343] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.952670 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:33.956389 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:33.957617 2684 raft_consensus.cc:491] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:33.957762 2684 raft_consensus.cc:513] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:33.958743 2684 leader_election.cc:290] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:33.958858 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:33.959481 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:33.959658 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:33.960395 986 leader_election.cc:343] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:33.960814 984 leader_election.cc:343] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:33.960939 984 leader_election.cc:304] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.961001 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:33.961349 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:33.964047 2684 raft_consensus.cc:2747] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:33.965030 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:33.965703 1540 catalog_manager.cc:5582] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:33.967523 1540 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:33.969216 1540 catalog_manager.cc:5582] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:33.973567 1522 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3] alter complete (version 1)
I20250905 19:52:33.975185 1542 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530] alter complete (version 1)
I20250905 19:52:33.975113 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.975776 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:33.975956 1037 raft_consensus.cc:2466] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:33.979176 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 89) took 4596 ms. Trace:
I20250905 19:52:33.979274 2379 rpcz_store.cc:276] 0905 19:52:29.382649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.382692 (+ 43us) service_pool.cc:225] Handling call
0905 19:52:33.772673 (+4389981us) ts_tablet_manager.cc:1145] Deleting tablet a5f076d9917f40d899d7422ad974fe0f
0905 19:52:33.772721 (+ 48us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.910648 (+137927us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.913862 (+ 3214us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.979153 (+ 65291us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4389777}
I20250905 19:52:33.979596 2379 tablet_replica.cc:331] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:33.979763 2379 raft_consensus.cc:2241] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:33.979919 2379 raft_consensus.cc:2270] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.979930 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a5f076d9917f40d899d7422ad974fe0f (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:33.980468 2379 ts_tablet_manager.cc:1905] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.986771 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 77) took 4627 ms. Trace:
I20250905 19:52:33.986912 2376 rpcz_store.cc:276] 0905 19:52:29.358792 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.359115 (+ 323us) service_pool.cc:225] Handling call
0905 19:52:33.648932 (+4289817us) ts_tablet_manager.cc:1145] Deleting tablet 091ef981d6b54756830927ea111e78aa
0905 19:52:33.648943 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.740532 (+ 91589us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.918796 (+178264us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.986701 (+ 67905us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4288056}
I20250905 19:52:33.987350 2376 tablet_replica.cc:331] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:33.987488 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 091ef981d6b54756830927ea111e78aa (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]) successfully deleted
I20250905 19:52:33.987597 2376 raft_consensus.cc:2241] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:33.988674 2823 tablet.cc:1722] T e44e88a0df444fe5b156a8fdf3199507 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.989955 2824 tablet.cc:1722] T efb70327f55844879f89449a7116010f P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:33.992147 2376 raft_consensus.cc:2270] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:33.995143 1170 tablet_bootstrap.cc:654] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:33.995839 2376 ts_tablet_manager.cc:1905] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:33.996263 849 leader_election.cc:304] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:33.997481 2422 raft_consensus.cc:2802] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:33.997660 2422 raft_consensus.cc:491] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:33.998837 2422 raft_consensus.cc:3058] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:33.998490 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.000981 1546 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet c14dc571b586405f8fd1eaa452998978
W20250905 19:52:34.001392 986 leader_election.cc:343] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.001528 986 leader_election.cc:304] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.001757 1546 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 51) took 5939 ms. Trace:
I20250905 19:52:34.001798 2684 raft_consensus.cc:2747] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.001900 1546 rpcz_store.cc:276] 0905 19:52:28.062411 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.884277 (+3821866us) service_pool.cc:225] Handling call
0905 19:52:31.913296 (+ 29019us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:31.913454 (+ 158us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:31.913525 (+ 71us) catalog_manager.cc:2799] Locking tablets
0905 19:52:31.913689 (+ 164us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:31.948134 (+ 34445us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:31.948299 (+ 165us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:34.000888 (+2052589us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:34.001708 (+ 820us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:31.914484 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:31.914602 (+ 118us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:31.914608 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:31.914611 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:31.914765 (+ 154us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:31.914789 (+ 24us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:31.914795 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:31.914944 (+ 149us) tablet.cc:699] Row locks acquired
0905 19:52:31.914947 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:31.914991 (+ 44us) write_op.cc:270] Start()
0905 19:52:31.915026 (+ 35us) write_op.cc:276] Timestamp: P: 1757101951914973 usec, L: 0
0905 19:52:31.915030 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:31.915221 (+ 191us) log.cc:844] Serialized 725 byte log entry
0905 19:52:31.919131 (+ 3910us) op_driver.cc:464] REPLICATION: finished
0905 19:52:31.921151 (+ 2020us) write_op.cc:301] APPLY: starting
0905 19:52:31.921171 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:31.921212 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:31.921247 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:31.921317 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:31.921352 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:31.921363 (+ 11us) write_op.cc:312] APPLY: finished
0905 19:52:31.921438 (+ 75us) log.cc:844] Serialized 29 byte log entry
0905 19:52:31.921499 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.921521 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:31.921528 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:31.921605 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.921609 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":1952,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":455,"num_ops":2,"prepare.queue_time_us":41,"prepare.run_cpu_time_us":843,"prepare.run_wall_time_us":843,"raft.queue_time_us":423,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":293,"replication_time_us":4068}]]}
I20250905 19:52:34.006551 849 leader_election.cc:304] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.007567 2565 raft_consensus.cc:2802] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.007725 2565 raft_consensus.cc:491] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.007839 2565 raft_consensus.cc:3058] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.008618 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c14dc571b586405f8fd1eaa452998978 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:31 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.008915 746 tablet_service.cc:1515] Processing DeleteTablet for tablet c14dc571b586405f8fd1eaa452998978 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:31 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:34.010953 877 tablet_service.cc:1515] Processing DeleteTablet for tablet c14dc571b586405f8fd1eaa452998978 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:31 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.011153 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.011307 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 8ed6109bdf8f4caf8c223a42401b94ab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.011476 875 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.011538 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.011745 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.012075 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.012270 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:34.013188 849 leader_election.cc:343] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:34.013319 849 leader_election.cc:304] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.014600 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.014987 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 05ae83eea5f74e9187713e8b82c9a23b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.015128 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 0267ac3e3f1a45978807737eca5d91ef with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.015170 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.015347 891 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.015568 2863 tablet.cc:1722] T e44e88a0df444fe5b156a8fdf3199507 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.015678 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.016088 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 78076f6534814939ae8c2db9a4bd1879 with delete_type TABLET_DATA_DELETED (replaced by tablet 53ac46ce05694bc2b53e195638be9e23) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.017558 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.018857 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.019068 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 2683a36998674f739b41d2a57339954c with delete_type TABLET_DATA_DELETED (replaced by tablet 3315d3e9633e42ea8d7a6b5934cb0438) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.019132 2864 tablet.cc:1722] T 0fb8fe93061848a0b055979bad10f08b P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.019829 2861 tablet.cc:1722] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.024055 2838 raft_consensus.cc:2747] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.029507 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 66) took 2459 ms. Trace:
I20250905 19:52:34.029618 2652 rpcz_store.cc:276] 0905 19:52:31.570321 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571001 (+ 680us) service_pool.cc:225] Handling call
0905 19:52:33.701348 (+2130347us) ts_tablet_manager.cc:1145] Deleting tablet 091ef981d6b54756830927ea111e78aa
0905 19:52:33.701361 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.816184 (+114823us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:33.911401 (+ 95217us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.029482 (+118081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2128106}
I20250905 19:52:34.030046 2379 ts_tablet_manager.cc:1918] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.030069 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 72) took 2458 ms. Trace:
I20250905 19:52:34.030150 2379 log.cc:1199] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/53ac46ce05694bc2b53e195638be9e23
I20250905 19:52:34.030135 2652 rpcz_store.cc:276] 0905 19:52:31.571865 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571926 (+ 61us) service_pool.cc:225] Handling call
0905 19:52:34.029920 (+2457994us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:34.029931 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.030052 (+ 121us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456336}
I20250905 19:52:34.030370 2652 tablet_replica.cc:331] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:34.030506 2379 ts_tablet_manager.cc:1939] T 53ac46ce05694bc2b53e195638be9e23 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.030540 2652 raft_consensus.cc:2241] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:34.030859 2652 raft_consensus.cc:2270] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.031364 2652 ts_tablet_manager.cc:1905] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.032051 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 091ef981d6b54756830927ea111e78aa (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]) successfully deleted
W20250905 19:52:34.032509 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 14d5eb8bec074d2c8766fe7b159031e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14d5eb8bec074d2c8766fe7b159031e5 already in progress: creating tablet
I20250905 19:52:34.034083 883 tablet_service.cc:1515] Processing DeleteTablet for tablet baefaa1f042348c1874e306f1e5a8781 with delete_type TABLET_DATA_DELETED (replaced by tablet 1e9b7458f96148b08962a9b82223187e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.038216 1169 tablet_bootstrap.cc:492] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.038389 1169 ts_tablet_manager.cc:1397] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.169s user 0.002s sys 0.003s
I20250905 19:52:34.040989 849 leader_election.cc:304] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.040774 1169 raft_consensus.cc:357] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.041131 1169 raft_consensus.cc:383] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.041095 883 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.041337 1169 raft_consensus.cc:738] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.041497 2838 raft_consensus.cc:2802] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.041608 2838 raft_consensus.cc:491] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.041695 2838 raft_consensus.cc:3058] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.042686 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.041931 1169 consensus_queue.cc:260] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.046557 1169 ts_tablet_manager.cc:1428] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:34.048225 2812 tablet.cc:1722] T 239539ac19124de192cadae494720a2b P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.048460 1170 tablet_bootstrap.cc:492] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.048803 1170 ts_tablet_manager.cc:1397] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.118s user 0.007s sys 0.002s
I20250905 19:52:34.049268 1171 tablet_bootstrap.cc:492] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.049422 1171 ts_tablet_manager.cc:1397] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.147s user 0.006s sys 0.007s
I20250905 19:52:34.050228 1171 raft_consensus.cc:357] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.050426 1171 raft_consensus.cc:383] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.050544 1171 raft_consensus.cc:738] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.050794 1171 consensus_queue.cc:260] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.050730 1170 raft_consensus.cc:357] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.051139 1170 raft_consensus.cc:383] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.051411 1169 tablet_bootstrap.cc:492] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.051609 1170 raft_consensus.cc:738] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.052317 1517 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e] alter complete (version 5)
I20250905 19:52:34.055833 849 leader_election.cc:304] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.056248 2738 raft_consensus.cc:2802] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.056371 2738 raft_consensus.cc:491] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.056514 2738 raft_consensus.cc:3058] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.065640 2832 tablet.cc:1722] T 0fb8fe93061848a0b055979bad10f08b P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.070665 2812 tablet.cc:1722] T ea6df8634a314ceabcf3fd004bf7e241 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:34.075142 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:34.081182 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8092952d755848ccaba3a2ad94b66dad" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.083062 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 90) took 4700 ms. Trace:
I20250905 19:52:34.083178 2379 rpcz_store.cc:276] 0905 19:52:29.383023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.383502 (+ 479us) service_pool.cc:225] Handling call
0905 19:52:33.979553 (+4596051us) ts_tablet_manager.cc:1145] Deleting tablet 53ac46ce05694bc2b53e195638be9e23
0905 19:52:33.979562 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:33.983596 (+ 4034us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.029928 (+ 46332us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.083038 (+ 53110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4595866}
I20250905 19:52:34.083521 2379 tablet_replica.cc:331] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.083750 2379 raft_consensus.cc:2241] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.083930 2379 raft_consensus.cc:2270] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.084623 2379 ts_tablet_manager.cc:1905] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.086530 2422 raft_consensus.cc:513] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.087879 2422 leader_election.cc:290] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.088346 1171 ts_tablet_manager.cc:1428] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.039s user 0.001s sys 0.001s
I20250905 19:52:34.088949 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d095804b51b4205a23ac3d3106312f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.089221 1171 tablet_bootstrap.cc:492] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:34.089880 848 leader_election.cc:343] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.092149 1171 tablet_bootstrap.cc:654] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.097958 712 leader_election.cc:304] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:34.098331 1849 raft_consensus.cc:2747] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.100373 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 53ac46ce05694bc2b53e195638be9e23 (table default.Table_a659dd4a6bcf4f618d1ddec59f33ffca [id=fcb432ba71c643d0b28d4f95bbfb4531]) successfully deleted
I20250905 19:52:34.101347 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:34.105844 2863 tablet.cc:1722] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:34.112939 1170 consensus_queue.cc:260] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.113991 1169 tablet_bootstrap.cc:654] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.114481 1170 ts_tablet_manager.cc:1428] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.066s user 0.003s sys 0.000s
I20250905 19:52:34.114696 1849 raft_consensus.cc:491] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.114830 1849 raft_consensus.cc:513] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.115242 1170 tablet_bootstrap.cc:492] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.115509 1849 leader_election.cc:290] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.125967 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.133641 771 raft_consensus.cc:3058] T 0e05db608a154b1ca0f9a1ed06c7fbd3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:34.136785 848 consensus_peers.cc:489] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:34.149286 1539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 75) took 5490 ms (client timeout 1000 ms). Trace:
W20250905 19:52:34.151623 848 consensus_peers.cc:489] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:34.155520 2413 raft_consensus.cc:491] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.155748 2413 raft_consensus.cc:513] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.156487 2413 leader_election.cc:290] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.157084 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bb79e3126074736a23b128a8e14fab6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.157656 848 leader_election.cc:343] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.158636 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1d095804b51b4205a23ac3d3106312f2" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.158814 1053 raft_consensus.cc:3058] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.162237 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1e9b7458f96148b08962a9b82223187e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.165530 2824 tablet.cc:1722] T 3315d3e9633e42ea8d7a6b5934cb0438 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.170117 2565 raft_consensus.cc:513] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.170831 2565 leader_election.cc:290] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.177523 774 raft_consensus.cc:3058] T 8092952d755848ccaba3a2ad94b66dad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.177695 1170 tablet_bootstrap.cc:654] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.180478 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:34.181255 848 leader_election.cc:343] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.161011 1536 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 74) took 6596 ms (client timeout 1000 ms). Trace:
W20250905 19:52:34.182492 1536 rpcz_store.cc:269] 0905 19:52:27.564622 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:27.899603 (+334981us) service_pool.cc:225] Handling call
0905 19:52:34.160930 (+6261327us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:28.921630 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:28.921834 (+ 204us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:28.921845 (+ 11us) write_op.cc:435] Acquired schema lock
0905 19:52:28.921850 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:28.922090 (+ 240us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:28.922104 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:28.922109 (+ 5us) tablet.cc:683] Acquiring locks for 4 operations
0905 19:52:28.922221 (+ 112us) tablet.cc:699] Row locks acquired
0905 19:52:28.922224 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:28.922268 (+ 44us) write_op.cc:270] Start()
0905 19:52:28.922314 (+ 46us) write_op.cc:276] Timestamp: P: 1757101948922247 usec, L: 0
0905 19:52:28.922319 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:28.923736 (+ 1417us) log.cc:844] Serialized 1650 byte log entry
0905 19:52:28.992298 (+ 68562us) op_driver.cc:464] REPLICATION: finished
0905 19:52:28.993716 (+ 1418us) write_op.cc:301] APPLY: starting
0905 19:52:28.993736 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:28.993799 (+ 63us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:28.993861 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:28.993992 (+ 131us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:28.994031 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0905 19:52:28.994039 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:28.994139 (+ 100us) log.cc:844] Serialized 41 byte log entry
0905 19:52:28.995945 (+ 1806us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.996004 (+ 59us) write_op.cc:454] Released schema lock
0905 19:52:28.996015 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:28.996120 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:28.996126 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":132475,"reported_tablets":188,"child_traces":[["op",{"apply.queue_time_us":1340,"apply.run_cpu_time_us":696,"apply.run_wall_time_us":2391,"num_ops":4,"prepare.queue_time_us":49623,"prepare.run_cpu_time_us":1134,"prepare.run_wall_time_us":2261,"raft.queue_time_us":38,"raft.run_cpu_time_us":6,"raft.run_wall_time_us":6,"replication_time_us":69930,"thread_start_us":52844,"threads_started":2}]]}
W20250905 19:52:34.179818 1539 rpcz_store.cc:269] 0905 19:52:28.659051 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.156349 (+497298us) service_pool.cc:225] Handling call
0905 19:52:34.149244 (+4992895us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:30.372415 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:30.372566 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:30.372573 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:30.372575 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:30.372723 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:30.372738 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:30.372751 (+ 13us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:30.372868 (+ 117us) tablet.cc:699] Row locks acquired
0905 19:52:30.372871 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:30.372906 (+ 35us) write_op.cc:270] Start()
0905 19:52:30.372940 (+ 34us) write_op.cc:276] Timestamp: P: 1757101950372885 usec, L: 0
0905 19:52:30.372944 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:30.373166 (+ 222us) log.cc:844] Serialized 901 byte log entry
0905 19:52:30.504290 (+131124us) op_driver.cc:464] REPLICATION: finished
0905 19:52:30.507951 (+ 3661us) write_op.cc:301] APPLY: starting
0905 19:52:30.508075 (+ 124us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:30.508126 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:30.508187 (+ 61us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:30.508277 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:30.508315 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:30.508322 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:30.508428 (+ 106us) log.cc:844] Serialized 29 byte log entry
0905 19:52:30.508986 (+ 558us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:30.509018 (+ 32us) write_op.cc:454] Released schema lock
0905 19:52:30.509030 (+ 12us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:30.539935 (+ 30905us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:30.539942 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:30.571458 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:30.571594 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:30.571601 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:30.571604 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:30.571752 (+ 148us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:30.571770 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:30.571779 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:30.571847 (+ 68us) tablet.cc:699] Row locks acquired
0905 19:52:30.571850 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:30.571896 (+ 46us) write_op.cc:270] Start()
0905 19:52:30.571934 (+ 38us) write_op.cc:276] Timestamp: P: 1757101950571871 usec, L: 0
0905 19:52:30.571944 (+ 10us) op_driver.cc:348] REPLICATION: starting
0905 19:52:30.572186 (+ 242us) log.cc:844] Serialized 356 byte log entry
0905 19:52:30.602224 (+ 30038us) op_driver.cc:464] REPLICATION: finished
0905 19:52:30.603222 (+ 998us) write_op.cc:301] APPLY: starting
0905 19:52:30.603244 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:30.603294 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:30.603339 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:30.603405 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:30.603442 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:30.603450 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:30.603537 (+ 87us) log.cc:844] Serialized 23 byte log entry
0905 19:52:30.603837 (+ 300us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:30.603870 (+ 33us) write_op.cc:454] Released schema lock
0905 19:52:30.603883 (+ 13us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:30.603994 (+ 111us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:30.604000 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":79641,"reported_tablets":198,"child_traces":[["op",{"apply.queue_time_us":3562,"apply.run_cpu_time_us":669,"apply.run_wall_time_us":31894,"num_ops":2,"prepare.queue_time_us":4540,"prepare.run_cpu_time_us":943,"prepare.run_wall_time_us":35999,"raft.queue_time_us":48,"raft.run_cpu_time_us":321,"raft.run_wall_time_us":321,"replication_time_us":131287,"thread_start_us":9840,"threads_started":2}],["op",{"apply.queue_time_us":931,"apply.run_cpu_time_us":527,"apply.run_wall_time_us":743,"mutex_wait_us":10723,"num_ops":1,"prepare.queue_time_us":25211,"prepare.run_cpu_time_us":819,"prepare.run_wall_time_us":820,"replication_time_us":30243}]]}
W20250905 19:52:34.187103 1539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 71) took 1089 ms (client timeout 1000 ms). Trace:
W20250905 19:52:34.187300 1539 rpcz_store.cc:269] 0905 19:52:33.097768 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.187006 (+1089238us) service_pool.cc:212] Skipping call since client already timed out
0905 19:52:34.187055 (+ 49us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:52:34.189594 1536 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_3c5a0a7b75424d2ab2975a941bab388b"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:34.191457 2565 consensus_queue.cc:1035] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.192642 2565 consensus_queue.cc:1035] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:34.194061 1536 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3c5a0a7b75424d2ab2975a941bab388b in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:34.194370 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.194371 2565 consensus_queue.cc:1035] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.195856 1171 tablet_bootstrap.cc:492] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.196059 1171 ts_tablet_manager.cc:1397] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.107s user 0.005s sys 0.004s
I20250905 19:52:34.196913 2789 raft_consensus.cc:491] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.196980 1171 raft_consensus.cc:357] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.197088 2789 raft_consensus.cc:513] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.197201 1171 raft_consensus.cc:383] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.197290 1171 raft_consensus.cc:738] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.197580 1171 consensus_queue.cc:260] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.197738 2789 leader_election.cc:290] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.198266 1171 ts_tablet_manager.cc:1428] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:34.198632 1171 tablet_bootstrap.cc:492] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.199681 881 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.199956 881 tablet_service.cc:1515] Processing DeleteTablet for tablet f28e403dc2f6471ebea1805fd843cfb1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.200143 878 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.200727 878 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.202822 769 raft_consensus.cc:3058] T 4dd2a96b8180438fbc722be1c18a0189 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.203446 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.204439 848 leader_election.cc:343] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.205595 878 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.207799 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 9193487b4ab74dafa76a386ddcaf674b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8bbbcac06944469e13f14ab1d80402) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.208015 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.208662 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.208693 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 6221d7d058944bdb9aa730791cf2417b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d19f0216a4f069237e48c62ee7875) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.208946 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.210367 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 06850faa25de4b3d83f6bf98ccd370f0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.210544 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.212961 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.213156 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.215519 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.216403 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.216660 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.217388 889 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.217782 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.217782 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ab4f69c45295422f85b310aa55220e9b with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.218416 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 5b0d4be788f841c2860f5dd65275b623 with delete_type TABLET_DATA_DELETED (replaced by tablet 37311372c03644b0a99ce92b5d42bfd1) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.218775 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.219112 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.219444 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.219771 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.219916 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bb79e3126074736a23b128a8e14fab6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.220141 889 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.220136 1051 raft_consensus.cc:2466] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.220342 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:34.220456 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 9e9994863c374354838aa0f817d0057e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.221171 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 54628965b82e470d811eff95ffc56c5f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.221526 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 3de936d2309a411eabbd59f7b73ce7f3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.221860 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.222328 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 6bb79e3126074736a23b128a8e14fab6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.223322 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 2350bd08f977424ba94ee8fa01f66efa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:34.224898 1170 tablet_bootstrap.cc:492] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.225183 1170 ts_tablet_manager.cc:1397] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.110s user 0.006s sys 0.000s
I20250905 19:52:34.227464 1170 raft_consensus.cc:357] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.228168 1170 raft_consensus.cc:383] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.228408 1170 raft_consensus.cc:738] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.228974 1170 consensus_queue.cc:260] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.230147 1170 ts_tablet_manager.cc:1428] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 19:52:34.230832 1170 tablet_bootstrap.cc:492] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.232528 775 raft_consensus.cc:3058] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.237316 1171 tablet_bootstrap.cc:654] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:34.230489 1532 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 65) took 4651 ms (client timeout 1000 ms). Trace:
W20250905 19:52:34.238127 1532 rpcz_store.cc:269] 0905 19:52:29.579420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.596859 (+ 17439us) service_pool.cc:225] Handling call
0905 19:52:34.230436 (+4633577us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:31.604479 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:31.604707 (+ 228us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:31.604714 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:31.604717 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:31.605587 (+ 870us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:31.605909 (+ 322us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:31.605916 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:31.606390 (+ 474us) tablet.cc:699] Row locks acquired
0905 19:52:31.606393 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:31.606899 (+ 506us) write_op.cc:270] Start()
0905 19:52:31.607239 (+ 340us) write_op.cc:276] Timestamp: P: 1757101951606717 usec, L: 0
0905 19:52:31.607244 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:31.609311 (+ 2067us) log.cc:844] Serialized 356 byte log entry
0905 19:52:31.764919 (+155608us) op_driver.cc:464] REPLICATION: finished
0905 19:52:31.803309 (+ 38390us) write_op.cc:301] APPLY: starting
0905 19:52:31.803439 (+ 130us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:31.803720 (+ 281us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:31.804014 (+ 294us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:31.805383 (+ 1369us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:31.805932 (+ 549us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:31.805946 (+ 14us) write_op.cc:312] APPLY: finished
0905 19:52:31.830041 (+ 24095us) log.cc:844] Serialized 23 byte log entry
0905 19:52:31.830100 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.830114 (+ 14us) write_op.cc:454] Released schema lock
0905 19:52:31.830119 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:31.830171 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:31.830174 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":35727,"reported_tablets":182,"child_traces":[["op",{"apply.queue_time_us":38074,"apply.run_cpu_time_us":789,"apply.run_wall_time_us":26982,"mutex_wait_us":710,"num_ops":1,"prepare.queue_time_us":5953,"prepare.run_cpu_time_us":1479,"prepare.run_wall_time_us":57347,"raft.queue_time_us":37,"raft.run_cpu_time_us":830,"raft.run_wall_time_us":830,"replication_time_us":112092,"spinlock_wait_cycles":16896,"thread_start_us":96314,"threads_started":3,"wal-append.queue_time_us":35348}]]}
I20250905 19:52:34.247216 1171 tablet_bootstrap.cc:492] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.247371 1171 ts_tablet_manager.cc:1397] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.001s
I20250905 19:52:34.248175 1171 raft_consensus.cc:357] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.248379 1171 raft_consensus.cc:383] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.248435 1171 raft_consensus.cc:738] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.248647 1171 consensus_queue.cc:260] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.249154 1171 ts_tablet_manager.cc:1428] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.252694 1171 tablet_bootstrap.cc:492] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.254300 1171 tablet_bootstrap.cc:654] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.255044 849 leader_election.cc:304] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.255594 2565 raft_consensus.cc:2802] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.255760 2565 raft_consensus.cc:491] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.255908 2565 raft_consensus.cc:3058] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.256949 712 leader_election.cc:304] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:34.257315 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.257387 1849 raft_consensus.cc:2747] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.257496 1051 raft_consensus.cc:3058] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.258031 2870 raft_consensus.cc:491] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.258198 2870 raft_consensus.cc:513] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.258816 2870 leader_election.cc:290] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.260198 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.260766 848 leader_election.cc:343] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.261356 1536 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 52) took 6166 ms. Trace:
I20250905 19:52:34.261847 2379 ts_tablet_manager.cc:1918] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.261953 2379 log.cc:1199] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/723168b075874dfcb3eb50c5134c499e
I20250905 19:52:34.262267 2379 ts_tablet_manager.cc:1939] T 723168b075874dfcb3eb50c5134c499e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.266366 2838 raft_consensus.cc:513] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.266983 2838 leader_election.cc:290] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.267282 1053 raft_consensus.cc:2466] T 1d095804b51b4205a23ac3d3106312f2 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.271066 2838 raft_consensus.cc:491] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.271209 2838 raft_consensus.cc:513] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.271868 2838 leader_election.cc:290] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.272553 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.273710 848 leader_election.cc:343] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.275231 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c14dc571b586405f8fd1eaa452998978" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:34.277887 848 leader_election.cc:343] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.281177 1171 tablet_bootstrap.cc:492] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.281397 1171 ts_tablet_manager.cc:1397] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.029s user 0.006s sys 0.000s
I20250905 19:52:34.283099 1171 raft_consensus.cc:357] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.283344 1171 raft_consensus.cc:383] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.283429 1171 raft_consensus.cc:738] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.283737 1171 consensus_queue.cc:260] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.284438 1171 ts_tablet_manager.cc:1428] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:34.284849 1171 tablet_bootstrap.cc:492] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.286756 1171 tablet_bootstrap.cc:654] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.294950 1171 tablet_bootstrap.cc:492] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.295154 1171 ts_tablet_manager.cc:1397] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.004s
I20250905 19:52:34.295876 1171 raft_consensus.cc:357] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.296097 1171 raft_consensus.cc:383] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.296180 1171 raft_consensus.cc:738] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.296471 1171 consensus_queue.cc:260] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.296998 1171 ts_tablet_manager.cc:1428] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.297312 1171 tablet_bootstrap.cc:492] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.297818 1849 raft_consensus.cc:491] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.297966 1849 raft_consensus.cc:513] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.298095 1170 tablet_bootstrap.cc:654] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.299072 1849 leader_election.cc:290] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:34.300670 711 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:34.302404 1171 tablet_bootstrap.cc:654] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.306861 1536 rpcz_store.cc:276] 0905 19:52:28.094564 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.189308 (+6094744us) service_pool.cc:225] Handling call
0905 19:52:34.246127 (+ 56819us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:34.246653 (+ 526us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:34.259043 (+ 12390us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:34.261138 (+ 2095us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:34.261287 (+ 149us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:34.247306 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:34.247423 (+ 117us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:34.247431 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:34.247434 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:34.247541 (+ 107us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:34.247557 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:34.247565 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:34.247649 (+ 84us) tablet.cc:699] Row locks acquired
0905 19:52:34.247651 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:34.247693 (+ 42us) write_op.cc:270] Start()
0905 19:52:34.247739 (+ 46us) write_op.cc:276] Timestamp: P: 1757101954247674 usec, L: 0
0905 19:52:34.247742 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:34.247954 (+ 212us) log.cc:844] Serialized 433 byte log entry
0905 19:52:34.251116 (+ 3162us) op_driver.cc:464] REPLICATION: finished
0905 19:52:34.251238 (+ 122us) write_op.cc:301] APPLY: starting
0905 19:52:34.251254 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:34.251306 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:34.251338 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:34.251471 (+ 133us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:34.251506 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:34.251513 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:34.251601 (+ 88us) log.cc:844] Serialized 29 byte log entry
0905 19:52:34.251703 (+ 102us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.251725 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:34.251733 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:34.251808 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.251812 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":71,"apply.run_cpu_time_us":562,"apply.run_wall_time_us":562,"num_ops":2,"prepare.queue_time_us":64,"prepare.run_cpu_time_us":790,"prepare.run_wall_time_us":790,"raft.queue_time_us":40,"raft.run_cpu_time_us":320,"raft.run_wall_time_us":319,"replication_time_us":3340}]]}
I20250905 19:52:34.310727 1169 tablet_bootstrap.cc:492] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.311162 1171 tablet_bootstrap.cc:492] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.311085 1169 ts_tablet_manager.cc:1397] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.261s user 0.006s sys 0.009s
I20250905 19:52:34.311291 1171 ts_tablet_manager.cc:1397] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.003s
I20250905 19:52:34.312009 1171 raft_consensus.cc:357] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.312178 1171 raft_consensus.cc:383] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.312232 1171 raft_consensus.cc:738] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.312417 1171 consensus_queue.cc:260] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.313273 1169 raft_consensus.cc:357] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.313795 1171 ts_tablet_manager.cc:1428] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.313782 1169 raft_consensus.cc:383] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.313931 1169 raft_consensus.cc:738] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.314196 1171 tablet_bootstrap.cc:492] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.314510 1169 consensus_queue.cc:260] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.315899 1171 tablet_bootstrap.cc:654] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.321889 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.324075 1049 raft_consensus.cc:2466] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.325335 1171 tablet_bootstrap.cc:492] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.325500 1171 ts_tablet_manager.cc:1397] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20250905 19:52:34.327445 1169 ts_tablet_manager.cc:1428] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.016s user 0.003s sys 0.000s
I20250905 19:52:34.327903 2684 raft_consensus.cc:491] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.328260 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.325295 1536 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_90ee22819c074028aaf36f24d7de572d" } modify_external_catalogs: true
I20250905 19:52:34.328260 2684 raft_consensus.cc:513] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.328650 1049 raft_consensus.cc:2466] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.329119 2684 leader_election.cc:290] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:34.330039 1536 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_90ee22819c074028aaf36f24d7de572d" } modify_external_catalogs: true
I20250905 19:52:34.330160 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.330628 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c14dc571b586405f8fd1eaa452998978" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
W20250905 19:52:34.330767 984 leader_election.cc:343] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.330850 1047 raft_consensus.cc:3058] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.326289 1171 raft_consensus.cc:357] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.329747 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.331003 1171 raft_consensus.cc:383] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.331218 1171 raft_consensus.cc:738] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:34.331383 986 leader_election.cc:343] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.331480 986 leader_election.cc:304] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.331449 1171 consensus_queue.cc:260] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.332036 1171 ts_tablet_manager.cc:1428] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:34.332036 2684 raft_consensus.cc:2747] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.332351 1171 tablet_bootstrap.cc:492] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.335431 1849 raft_consensus.cc:491] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.335582 1849 raft_consensus.cc:513] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.336606 1849 leader_election.cc:290] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.337371 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.338249 1171 tablet_bootstrap.cc:654] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.346011 1170 tablet_bootstrap.cc:492] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.346025 1169 tablet_bootstrap.cc:492] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.346153 1170 ts_tablet_manager.cc:1397] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.116s user 0.005s sys 0.000s
I20250905 19:52:34.346899 1170 raft_consensus.cc:357] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.347059 1170 raft_consensus.cc:383] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.347106 1170 raft_consensus.cc:738] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.347280 1170 consensus_queue.cc:260] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.347745 1169 tablet_bootstrap.cc:654] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.347745 1170 ts_tablet_manager.cc:1428] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:34.348117 1170 tablet_bootstrap.cc:492] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.352169 1170 tablet_bootstrap.cc:654] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.360158 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.361460 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 91) took 4977 ms. Trace:
I20250905 19:52:34.361559 2379 rpcz_store.cc:276] 0905 19:52:29.383844 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.383886 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:34.083472 (+4699586us) ts_tablet_manager.cc:1145] Deleting tablet 723168b075874dfcb3eb50c5134c499e
0905 19:52:34.083483 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.168927 (+ 85444us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.261740 (+ 92813us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.361438 (+ 99698us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4699402}
I20250905 19:52:34.361955 2379 tablet_replica.cc:331] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.362165 2379 raft_consensus.cc:2241] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.362332 2379 raft_consensus.cc:2270] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.362951 2379 ts_tablet_manager.cc:1905] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.369899 2738 raft_consensus.cc:513] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.370554 2738 leader_election.cc:290] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.377509 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4d6f2e44855d4312b1c0367145e3cee8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.378077 849 leader_election.cc:304] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.379000 2789 raft_consensus.cc:2802] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.379873 2568 consensus_queue.cc:1035] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.380558 2568 consensus_queue.cc:1035] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.380779 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 723168b075874dfcb3eb50c5134c499e (table default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]) successfully deleted
I20250905 19:52:34.381263 2789 raft_consensus.cc:695] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.381541 2789 consensus_queue.cc:237] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.381958 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:34.383250 1037 raft_consensus.cc:2466] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.371167 1051 raft_consensus.cc:2466] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
W20250905 19:52:34.384541 848 consensus_peers.cc:489] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:34.385067 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:34.386445 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:34.387329 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:34.390730 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:34.391362 848 leader_election.cc:343] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.395140 1169 tablet_bootstrap.cc:492] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.395295 1169 ts_tablet_manager.cc:1397] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.066s user 0.000s sys 0.003s
I20250905 19:52:34.396251 1169 raft_consensus.cc:357] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.396435 1169 raft_consensus.cc:383] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.396502 1169 raft_consensus.cc:738] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.396715 1169 consensus_queue.cc:260] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.397169 1169 ts_tablet_manager.cc:1428] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.397583 1169 tablet_bootstrap.cc:492] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.399619 1169 tablet_bootstrap.cc:654] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.410686 1171 tablet_bootstrap.cc:492] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.410866 1171 ts_tablet_manager.cc:1397] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.005s
I20250905 19:52:34.412432 1171 raft_consensus.cc:357] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.412663 1171 raft_consensus.cc:383] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.412719 1171 raft_consensus.cc:738] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.412950 1171 consensus_queue.cc:260] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.413487 1171 ts_tablet_manager.cc:1428] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:34.413764 1171 tablet_bootstrap.cc:492] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.415210 1171 tablet_bootstrap.cc:654] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.425423 1171 tablet_bootstrap.cc:492] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.425491 1169 tablet_bootstrap.cc:492] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.425607 1171 ts_tablet_manager.cc:1397] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.002s
I20250905 19:52:34.425618 1169 ts_tablet_manager.cc:1397] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.028s user 0.000s sys 0.003s
I20250905 19:52:34.413533 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:34.426338 1171 raft_consensus.cc:357] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.426518 1171 raft_consensus.cc:383] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250905 19:52:34.426482 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 47280.3us
I20250905 19:52:34.426573 1171 raft_consensus.cc:738] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.426724 849 leader_election.cc:304] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.426949 2422 raft_consensus.cc:2802] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.427062 2422 raft_consensus.cc:491] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.427145 849 leader_election.cc:304] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.427201 2422 raft_consensus.cc:3058] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.427452 2568 raft_consensus.cc:2802] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.427536 2568 raft_consensus.cc:491] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.427595 2568 raft_consensus.cc:3058] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.427671 2738 consensus_queue.cc:1035] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.428323 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:34.426766 1171 consensus_queue.cc:260] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.429795 2738 consensus_queue.cc:1035] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.430291 1170 tablet_bootstrap.cc:492] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.430446 1170 ts_tablet_manager.cc:1397] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.083s user 0.002s sys 0.002s
I20250905 19:52:34.430366 1169 raft_consensus.cc:357] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.430562 1169 raft_consensus.cc:383] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.430615 1169 raft_consensus.cc:738] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.430823 1169 consensus_queue.cc:260] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.431241 849 leader_election.cc:304] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.431249 1170 raft_consensus.cc:357] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.431342 1169 ts_tablet_manager.cc:1428] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 19:52:34.431416 1170 raft_consensus.cc:383] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.431478 1170 raft_consensus.cc:738] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.431628 1169 tablet_bootstrap.cc:492] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.431667 2738 raft_consensus.cc:2802] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.431738 2738 raft_consensus.cc:491] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.431809 2738 raft_consensus.cc:3058] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.432008 1170 consensus_queue.cc:260] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.432374 849 leader_election.cc:304] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.432859 2838 raft_consensus.cc:2802] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.432971 2838 raft_consensus.cc:695] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.433199 2838 consensus_queue.cc:237] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.433372 1170 ts_tablet_manager.cc:1428] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250905 19:52:34.433990 1170 tablet_bootstrap.cc:492] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.435178 2812 tablet.cc:1722] T 0fb8fe93061848a0b055979bad10f08b P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.440671 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.466560 1849 raft_consensus.cc:491] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.466753 1849 raft_consensus.cc:513] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.467384 1849 leader_election.cc:290] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.467520 2376 ts_tablet_manager.cc:1918] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.467573 1169 tablet_bootstrap.cc:654] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.467624 2376 log.cc:1199] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ba161282480e4736889559a2e37b195f
I20250905 19:52:34.467944 2376 ts_tablet_manager.cc:1939] T ba161282480e4736889559a2e37b195f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:34.468230 1047 raft_consensus.cc:2466] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.468777 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.470880 2379 ts_tablet_manager.cc:1918] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.470976 2379 log.cc:1199] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a3805127dd78489e8545da7414ead904
I20250905 19:52:34.471570 2565 raft_consensus.cc:513] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.472224 2565 leader_election.cc:290] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.473660 2652 ts_tablet_manager.cc:1918] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.473757 2652 log.cc:1199] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/1f1a8e7059b348cba34069ef49bf1aad
I20250905 19:52:34.474118 2652 ts_tablet_manager.cc:1939] T 1f1a8e7059b348cba34069ef49bf1aad P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:34.476940 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 69) took 2906 ms. Trace:
I20250905 19:52:34.477047 2652 rpcz_store.cc:276] 0905 19:52:31.570703 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.571183 (+ 480us) service_pool.cc:225] Handling call
0905 19:52:34.030331 (+2459148us) ts_tablet_manager.cc:1145] Deleting tablet 1f1a8e7059b348cba34069ef49bf1aad
0905 19:52:34.030339 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.261731 (+231392us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.473575 (+211844us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.476915 (+ 3340us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2456597}
I20250905 19:52:34.477416 2652 tablet_replica.cc:331] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:34.477595 2652 raft_consensus.cc:2241] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.477742 2652 raft_consensus.cc:2270] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.478269 2652 ts_tablet_manager.cc:1905] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.480291 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 1f1a8e7059b348cba34069ef49bf1aad (table default.Table_7186efc2caef4940ade686830c963ba1 [id=0a06a88771a94e60b5bb57fa7f7d8215]) successfully deleted
I20250905 19:52:34.484664 1170 tablet_bootstrap.cc:654] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.488893 2568 raft_consensus.cc:513] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.489575 2568 leader_election.cc:290] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.492180 2738 raft_consensus.cc:513] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.492772 2738 leader_election.cc:290] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.493177 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 78) took 5134 ms. Trace:
I20250905 19:52:34.493281 2376 rpcz_store.cc:276] 0905 19:52:29.358905 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.359567 (+ 662us) service_pool.cc:225] Handling call
0905 19:52:33.987259 (+4627692us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:33.987269 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.365977 (+378708us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.467414 (+101437us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.493159 (+ 25745us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":7145,"tablet-delete.queue_time_us":4626200}
I20250905 19:52:34.493680 2376 tablet_replica.cc:331] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:34.486129 2422 raft_consensus.cc:513] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.495237 1169 tablet_bootstrap.cc:492] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.495379 1169 ts_tablet_manager.cc:1397] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.064s user 0.004s sys 0.004s
I20250905 19:52:34.496124 1169 raft_consensus.cc:357] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.496285 1169 raft_consensus.cc:383] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.496338 1169 raft_consensus.cc:738] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.429073 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:34.496536 1169 consensus_queue.cc:260] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.496757 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:34.497033 1169 ts_tablet_manager.cc:1428] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.497079 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:34.497311 1169 tablet_bootstrap.cc:492] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.497812 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:34.498016 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:34.497869 2422 leader_election.cc:290] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.498404 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:34.498752 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:34.499290 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
I20250905 19:52:34.499536 1169 tablet_bootstrap.cc:654] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.499629 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:34.499861 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
W20250905 19:52:34.500630 511 leader_election.cc:336] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687): Timed out: RequestConsensusVote RPC to 127.0.104.62:38687 timed out after 0.564s (SENT)
I20250905 19:52:34.500794 511 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:34.502735 2652 ts_tablet_manager.cc:1918] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.508066 2652 log.cc:1199] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:34.508570 2652 ts_tablet_manager.cc:1939] T c0ba1d4a7cd444c688d5a9bc8fd1a0ff P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:34.513290 2870 raft_consensus.cc:491] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.513500 2870 raft_consensus.cc:513] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.514099 2870 leader_election.cc:290] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.516647 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bb79e3126074736a23b128a8e14fab6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.519683 2376 raft_consensus.cc:2241] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.519882 2376 raft_consensus.cc:2270] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.520117 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.520484 2376 ts_tablet_manager.cc:1905] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.522114 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.523576 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.526219 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.526641 2907 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20250905 19:52:34.528716 2376 ts_tablet_manager.cc:1918] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.528827 2376 log.cc:1199] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:34.529218 2376 ts_tablet_manager.cc:1939] T 6f3473785fff45d692bd50b1015ffe39 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:34.530063 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 76) took 5171 ms. Trace:
I20250905 19:52:34.530157 2376 rpcz_store.cc:276] 0905 19:52:29.358626 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.359161 (+ 535us) service_pool.cc:225] Handling call
0905 19:52:34.493631 (+5134470us) ts_tablet_manager.cc:1145] Deleting tablet 6f3473785fff45d692bd50b1015ffe39
0905 19:52:34.493641 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.525289 (+ 31648us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.528623 (+ 3334us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.530044 (+ 1421us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":25785,"tablet-delete.queue_time_us":5132539}
I20250905 19:52:34.530483 2376 tablet_replica.cc:331] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:34.530295 848 leader_election.cc:343] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.530740 2376 raft_consensus.cc:2241] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:34.531311 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ba161282480e4736889559a2e37b195f (table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]) successfully deleted
I20250905 19:52:34.531469 2789 raft_consensus.cc:491] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.531605 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6f3473785fff45d692bd50b1015ffe39 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]) successfully deleted
I20250905 19:52:34.531581 2789 raft_consensus.cc:513] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.531757 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
W20250905 19:52:34.531104 848 leader_election.cc:343] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.532153 2789 leader_election.cc:290] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:34.532336 848 leader_election.cc:343] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.532306 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]). Expected version 2 got 0
W20250905 19:52:34.532663 848 leader_election.cc:343] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.533008 848 consensus_peers.cc:489] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:34.533373 848 leader_election.cc:343] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.534682 712 leader_election.cc:343] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:34.534804 712 leader_election.cc:304] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:34.535168 712 leader_election.cc:304] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:34.535458 2888 raft_consensus.cc:2747] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.535596 2888 raft_consensus.cc:2747] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.537189 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f034b8cb5d8943938cb2b58c33b45494" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.539592 1170 tablet_bootstrap.cc:492] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.539724 1170 ts_tablet_manager.cc:1397] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.106s user 0.003s sys 0.001s
I20250905 19:52:34.540449 1170 raft_consensus.cc:357] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.540606 1170 raft_consensus.cc:383] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.540663 1170 raft_consensus.cc:738] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.540855 1170 consensus_queue.cc:260] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.541369 1170 ts_tablet_manager.cc:1428] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.541620 1170 tablet_bootstrap.cc:492] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.542969 1170 tablet_bootstrap.cc:654] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.549837 2870 raft_consensus.cc:491] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.549957 2870 raft_consensus.cc:513] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.550419 2870 leader_election.cc:290] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:34.551749 848 leader_election.cc:343] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.559607 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.560518 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:34.561097 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:34.561726 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:34.562288 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:34.562763 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
W20250905 19:52:34.562635 848 leader_election.cc:343] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.563516 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:34.563839 1546 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:34.571682 1170 tablet_bootstrap.cc:492] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.571849 1170 ts_tablet_manager.cc:1397] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.001s
I20250905 19:52:34.572603 1170 raft_consensus.cc:357] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.572793 1170 raft_consensus.cc:383] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.572849 1170 raft_consensus.cc:738] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.573069 1170 consensus_queue.cc:260] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.573596 1170 ts_tablet_manager.cc:1428] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.573882 1170 tablet_bootstrap.cc:492] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.575891 1170 tablet_bootstrap.cc:654] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.577965 2870 raft_consensus.cc:491] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.578109 2870 raft_consensus.cc:513] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.578697 2870 leader_election.cc:290] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.580700 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.582167 848 leader_election.cc:343] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.582440 2893 log.cc:927] Time spent T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.127s user 0.000s sys 0.000s
W20250905 19:52:34.584329 2890 log.cc:927] Time spent T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.130s user 0.000s sys 0.000s
I20250905 19:52:34.589942 775 raft_consensus.cc:3058] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.549270 1169 tablet_bootstrap.cc:492] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.579564 2496 consensus_queue.cc:1035] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:34.608553 1169 ts_tablet_manager.cc:1397] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.111s user 0.005s sys 0.000s
I20250905 19:52:34.608639 1170 tablet_bootstrap.cc:492] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.608783 1170 ts_tablet_manager.cc:1397] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.000s
I20250905 19:52:34.586036 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6bb79e3126074736a23b128a8e14fab6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.609078 1044 raft_consensus.cc:3058] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.609395 1169 raft_consensus.cc:357] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.609464 1170 raft_consensus.cc:357] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.609568 1169 raft_consensus.cc:383] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.609627 1169 raft_consensus.cc:738] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.609627 1170 raft_consensus.cc:383] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.609681 1170 raft_consensus.cc:738] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.609792 1169 consensus_queue.cc:260] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.609866 1170 consensus_queue.cc:260] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.610241 1169 ts_tablet_manager.cc:1428] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:34.610342 1170 ts_tablet_manager.cc:1428] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:34.611727 1044 raft_consensus.cc:2466] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.587586 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.612121 1036 raft_consensus.cc:3058] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.613734 2379 ts_tablet_manager.cc:1939] T a3805127dd78489e8545da7414ead904 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.614665 1036 raft_consensus.cc:2466] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.589257 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.614903 1045 raft_consensus.cc:3058] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.616711 2812 tablet.cc:1722] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:34.617563 1045 raft_consensus.cc:2466] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.589486 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.617791 1041 raft_consensus.cc:3058] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.620339 1041 raft_consensus.cc:2466] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.621856 1169 tablet_bootstrap.cc:492] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.622481 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 74) took 3048 ms. Trace:
I20250905 19:52:34.622985 1170 tablet_bootstrap.cc:492] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.628100 1170 tablet_bootstrap.cc:654] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.640918 1849 raft_consensus.cc:491] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.641057 1849 raft_consensus.cc:513] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.641670 1849 leader_election.cc:290] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.643735 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 92) took 5259 ms. Trace:
I20250905 19:52:34.643846 2379 rpcz_store.cc:276] 0905 19:52:29.384227 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.384267 (+ 40us) service_pool.cc:225] Handling call
0905 19:52:34.361882 (+4977615us) ts_tablet_manager.cc:1145] Deleting tablet a3805127dd78489e8545da7414ead904
0905 19:52:34.361893 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.467042 (+105149us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.470784 (+ 3742us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.643714 (+172930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4977432}
I20250905 19:52:34.644258 2379 tablet_replica.cc:331] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.644397 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a3805127dd78489e8545da7414ead904 (table default.Table_e65ca281d8804277a83ab7bfc2ab9cca [id=3ed6b530762e4290aa5c45ee1a55c100]) successfully deleted
I20250905 19:52:34.644481 2379 raft_consensus.cc:2241] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.644657 2379 raft_consensus.cc:2270] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.645289 2379 ts_tablet_manager.cc:1905] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.645882 2376 raft_consensus.cc:2270] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.646405 2376 ts_tablet_manager.cc:1905] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.623482 1169 tablet_bootstrap.cc:654] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.626304 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.627599 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f034b8cb5d8943938cb2b58c33b45494" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.628059 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.635911 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.640110 2496 raft_consensus.cc:491] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.642570 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "8d38452d3a0641ffa4af5d935f8ad9f0" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:34.642575 984 consensus_peers.cc:489] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:34.651777 1045 raft_consensus.cc:2466] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.651854 1036 raft_consensus.cc:2466] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.652012 1044 raft_consensus.cc:2466] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.652243 1046 raft_consensus.cc:2391] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:34.652138 2496 raft_consensus.cc:513] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.654079 2496 leader_election.cc:290] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:34.654901 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.654994 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.655117 909 raft_consensus.cc:2466] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
W20250905 19:52:34.655575 984 leader_election.cc:343] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.655824 986 leader_election.cc:304] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.656263 2496 raft_consensus.cc:2802] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.656363 2496 raft_consensus.cc:491] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.656450 2496 raft_consensus.cc:3058] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.656703 1170 tablet_bootstrap.cc:492] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.656843 1170 ts_tablet_manager.cc:1397] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.008s
I20250905 19:52:34.657580 1170 raft_consensus.cc:357] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.657740 1170 raft_consensus.cc:383] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.657801 1170 raft_consensus.cc:738] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.658007 1170 consensus_queue.cc:260] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.658490 1170 ts_tablet_manager.cc:1428] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:34.663275 2496 raft_consensus.cc:513] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.664215 2496 leader_election.cc:290] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:34.664662 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:34.664908 909 raft_consensus.cc:3058] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.664933 1041 raft_consensus.cc:2391] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 835daf30e4324782848290be181ae78f in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:34.665211 1170 tablet_bootstrap.cc:492] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.665500 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:34.665999 984 leader_election.cc:343] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.666769 1170 tablet_bootstrap.cc:654] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.651520 2652 rpcz_store.cc:276] 0905 19:52:31.573523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.573785 (+ 262us) service_pool.cc:225] Handling call
0905 19:52:34.477367 (+2903582us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:34.477378 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.499225 (+ 21847us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.502637 (+ 3412us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.622456 (+119819us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2903439}
I20250905 19:52:34.667446 2652 tablet_replica.cc:331] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:34.670697 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff (table default.Table_1c6571236cd7434d98aec149d06dc6b8 [id=6709b02c192c4ff88715b1f3b886b222]) successfully deleted
I20250905 19:52:34.671579 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e44e88a0df444fe5b156a8fdf3199507" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.672094 909 raft_consensus.cc:2466] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:34.672657 986 leader_election.cc:304] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.673200 2496 raft_consensus.cc:2802] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.673472 2496 raft_consensus.cc:695] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:34.673770 2496 consensus_queue.cc:237] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.676404 1170 tablet_bootstrap.cc:492] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.676718 1170 ts_tablet_manager.cc:1397] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.001s
I20250905 19:52:34.678473 1170 raft_consensus.cc:357] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.678658 1170 raft_consensus.cc:383] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.678718 1170 raft_consensus.cc:738] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.678900 1170 consensus_queue.cc:260] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:34.679908 848 consensus_peers.cc:489] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:34.667646 2652 raft_consensus.cc:2241] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.681994 2652 raft_consensus.cc:2270] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.682739 2652 ts_tablet_manager.cc:1905] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.684453 1170 ts_tablet_manager.cc:1428] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:34.684880 1170 tablet_bootstrap.cc:492] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.685295 2413 raft_consensus.cc:491] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.685513 2413 raft_consensus.cc:513] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.686230 2413 leader_election.cc:290] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.687103 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.687436 2379 ts_tablet_manager.cc:1918] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:34.687666 2379 log.cc:1199] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/2d84653c9e7a4857a7580fa85f8e4732
W20250905 19:52:34.688103 848 leader_election.cc:343] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.688676 1170 tablet_bootstrap.cc:654] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.688930 849 leader_election.cc:304] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.689219 2838 raft_consensus.cc:2802] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.689234 2379 ts_tablet_manager.cc:1939] T 2d84653c9e7a4857a7580fa85f8e4732 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.689379 2838 raft_consensus.cc:695] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.689688 2838 consensus_queue.cc:237] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.689980 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 93) took 5305 ms. Trace:
I20250905 19:52:34.690073 2379 rpcz_store.cc:276] 0905 19:52:29.384591 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.384628 (+ 37us) service_pool.cc:225] Handling call
0905 19:52:34.644199 (+5259571us) ts_tablet_manager.cc:1145] Deleting tablet 2d84653c9e7a4857a7580fa85f8e4732
0905 19:52:34.644211 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.648803 (+ 4592us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.687160 (+ 38357us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.689961 (+ 2801us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5259419}
I20250905 19:52:34.693801 2379 tablet_replica.cc:331] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.693951 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 2d84653c9e7a4857a7580fa85f8e4732 (table default.Table_48c2edbf02444d7580fff76f96c9c655 [id=12633ed634a44d74830d55d4e7191808]) successfully deleted
I20250905 19:52:34.694006 2379 raft_consensus.cc:2241] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:34.694270 2379 raft_consensus.cc:2270] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.697446 2379 ts_tablet_manager.cc:1905] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.709539 1171 ts_tablet_manager.cc:1428] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.284s user 0.001s sys 0.001s
I20250905 19:52:34.709697 2379 ts_tablet_manager.cc:1918] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.709818 2379 log.cc:1199] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:34.710134 2901 raft_consensus.cc:491] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.710392 1171 tablet_bootstrap.cc:492] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.710317 2901 raft_consensus.cc:513] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.712447 2901 leader_election.cc:290] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.713104 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.715200 848 leader_election.cc:343] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.715667 2379 ts_tablet_manager.cc:1939] T 6f3473785fff45d692bd50b1015ffe39 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.716465 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 94) took 5331 ms. Trace:
I20250905 19:52:34.716560 2379 rpcz_store.cc:276] 0905 19:52:29.384916 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.385235 (+ 319us) service_pool.cc:225] Handling call
0905 19:52:34.693754 (+5308519us) ts_tablet_manager.cc:1145] Deleting tablet 6f3473785fff45d692bd50b1015ffe39
0905 19:52:34.693765 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.706472 (+ 12707us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.709583 (+ 3111us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.716442 (+ 6859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5308318}
I20250905 19:52:34.717134 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6f3473785fff45d692bd50b1015ffe39 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]) successfully deleted
I20250905 19:52:34.718650 2379 tablet_replica.cc:331] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.718861 2379 raft_consensus.cc:2241] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.718856 849 leader_election.cc:304] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.719027 2379 raft_consensus.cc:2270] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.719581 2379 ts_tablet_manager.cc:1905] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.724489 849 leader_election.cc:304] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.741153 2902 raft_consensus.cc:2802] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.741305 2902 raft_consensus.cc:695] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.741551 2902 consensus_queue.cc:237] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.746385 2379 ts_tablet_manager.cc:1918] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.746582 2379 log.cc:1199] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/091ef981d6b54756830927ea111e78aa
I20250905 19:52:34.746961 2379 ts_tablet_manager.cc:1939] T 091ef981d6b54756830927ea111e78aa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.747870 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 96) took 5359 ms. Trace:
I20250905 19:52:34.748013 2379 rpcz_store.cc:276] 0905 19:52:29.388187 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.392220 (+ 4033us) service_pool.cc:225] Handling call
0905 19:52:34.716857 (+5324637us) ts_tablet_manager.cc:1145] Deleting tablet 091ef981d6b54756830927ea111e78aa
0905 19:52:34.718583 (+ 1726us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.722685 (+ 4102us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.735465 (+ 12780us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.747843 (+ 12378us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5324410}
I20250905 19:52:34.724805 2901 raft_consensus.cc:2802] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.731925 2789 raft_consensus.cc:491] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.748425 2379 tablet_replica.cc:331] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.748605 2379 raft_consensus.cc:2241] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.748610 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 091ef981d6b54756830927ea111e78aa (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]) successfully deleted
I20250905 19:52:34.748767 2379 raft_consensus.cc:2270] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.748585 2789 raft_consensus.cc:513] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.749387 2379 ts_tablet_manager.cc:1905] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.749413 2789 leader_election.cc:290] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.750375 2901 raft_consensus.cc:695] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.750829 2901 consensus_queue.cc:237] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.755165 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "421b28d047124222ae9cbc603ec22cee" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:34.758589 848 leader_election.cc:343] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.758968 1169 tablet_bootstrap.cc:492] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.759347 1169 ts_tablet_manager.cc:1397] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.138s user 0.013s sys 0.002s
I20250905 19:52:34.761657 1169 raft_consensus.cc:357] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.762162 1169 raft_consensus.cc:383] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.762408 1169 raft_consensus.cc:738] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.763109 1169 consensus_queue.cc:260] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.761834 849 leader_election.cc:304] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.770818 849 leader_election.cc:304] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.771642 2902 raft_consensus.cc:2802] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.771809 2902 raft_consensus.cc:695] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.772150 2902 consensus_queue.cc:237] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.773509 2838 raft_consensus.cc:2802] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:34.774029 2838 raft_consensus.cc:695] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:34.774298 2838 consensus_queue.cc:237] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.774928 2379 ts_tablet_manager.cc:1918] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.775055 2379 log.cc:1199] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/44769f1170354870ae2d34fdb67ffbae
I20250905 19:52:34.775427 2379 ts_tablet_manager.cc:1939] T 44769f1170354870ae2d34fdb67ffbae P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.776229 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 97) took 5383 ms. Trace:
I20250905 19:52:34.776324 2379 rpcz_store.cc:276] 0905 19:52:29.392684 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.393044 (+ 360us) service_pool.cc:225] Handling call
0905 19:52:34.748367 (+5355323us) ts_tablet_manager.cc:1145] Deleting tablet 44769f1170354870ae2d34fdb67ffbae
0905 19:52:34.748379 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.761063 (+ 12684us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.774810 (+ 13747us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.776209 (+ 1399us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5355152}
I20250905 19:52:34.776666 2379 tablet_replica.cc:331] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.776851 2379 raft_consensus.cc:2241] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.776839 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 44769f1170354870ae2d34fdb67ffbae (table default.Table_885d880838c44eab9ff1220768e59373 [id=3235ea3e87b249e884b027b840d4fd77]) successfully deleted
I20250905 19:52:34.776988 2379 raft_consensus.cc:2270] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.777513 2379 ts_tablet_manager.cc:1905] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.782171 1171 tablet_bootstrap.cc:654] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.787050 2376 ts_tablet_manager.cc:1918] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.794198 1169 ts_tablet_manager.cc:1428] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.035s user 0.003s sys 0.001s
I20250905 19:52:34.795027 1169 tablet_bootstrap.cc:492] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:34.797541 2379 ts_tablet_manager.cc:1918] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.797672 2379 log.cc:1199] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/c46ea498d6a14d1ebdffea415cfc7114
I20250905 19:52:34.798043 2379 ts_tablet_manager.cc:1939] T c46ea498d6a14d1ebdffea415cfc7114 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.798401 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 98) took 5405 ms. Trace:
I20250905 19:52:34.798492 2379 rpcz_store.cc:276] 0905 19:52:29.393420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.393660 (+ 240us) service_pool.cc:225] Handling call
0905 19:52:34.776619 (+5382959us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:34.776631 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.787016 (+ 10385us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.797412 (+ 10396us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.798372 (+ 960us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5382822}
I20250905 19:52:34.798847 2379 tablet_replica.cc:331] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.799068 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet c46ea498d6a14d1ebdffea415cfc7114 (table default.Table_f542b700d5364bfeb6e1e0111834fab0 [id=27f7071a498d467aa9047b07dac76b64]) successfully deleted
I20250905 19:52:34.799906 1170 tablet_bootstrap.cc:492] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.800072 1170 ts_tablet_manager.cc:1397] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.115s user 0.016s sys 0.018s
I20250905 19:52:34.800777 1170 raft_consensus.cc:357] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.801021 1170 raft_consensus.cc:383] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.801083 1170 raft_consensus.cc:738] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.801319 1170 consensus_queue.cc:260] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.801805 1170 ts_tablet_manager.cc:1428] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:34.802083 1170 tablet_bootstrap.cc:492] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.804351 1170 tablet_bootstrap.cc:654] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.799028 2379 raft_consensus.cc:2241] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.805321 2379 raft_consensus.cc:2270] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.805920 2379 ts_tablet_manager.cc:1905] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.810461 1051 raft_consensus.cc:3058] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.812373 2652 ts_tablet_manager.cc:1918] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.791710 2901 raft_consensus.cc:491] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.812721 2901 raft_consensus.cc:513] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.813676 2901 leader_election.cc:290] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.795895 849 leader_election.cc:304] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.801585 2413 raft_consensus.cc:491] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.814724 2413 raft_consensus.cc:513] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.815320 2413 leader_election.cc:290] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.815508 2413 raft_consensus.cc:2802] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.815626 2413 raft_consensus.cc:491] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.815745 2413 raft_consensus.cc:3058] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.815913 1170 tablet_bootstrap.cc:492] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.817282 1170 ts_tablet_manager.cc:1397] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.001s
I20250905 19:52:34.817972 1170 raft_consensus.cc:357] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.818137 1170 raft_consensus.cc:383] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.818205 1170 raft_consensus.cc:738] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.818439 1170 consensus_queue.cc:260] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.818925 1170 ts_tablet_manager.cc:1428] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:34.819348 1170 tablet_bootstrap.cc:492] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.819270 2413 raft_consensus.cc:513] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.819706 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:34.820349 2413 leader_election.cc:290] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.820791 849 leader_election.cc:304] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.821038 1170 tablet_bootstrap.cc:654] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.821277 849 leader_election.cc:304] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.821610 1171 tablet_bootstrap.cc:492] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.821728 849 leader_election.cc:304] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.821992 2379 ts_tablet_manager.cc:1918] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.822100 2379 log.cc:1199] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8503f6918ffb44efa36214633c93e31d
I20250905 19:52:34.822433 2789 raft_consensus.cc:2802] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.822468 2379 ts_tablet_manager.cc:1939] T 8503f6918ffb44efa36214633c93e31d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.822513 2789 raft_consensus.cc:491] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.822583 2789 raft_consensus.cc:3058] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.822778 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 99) took 5428 ms. Trace:
I20250905 19:52:34.822860 2379 rpcz_store.cc:276] 0905 19:52:29.394011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.394302 (+ 291us) service_pool.cc:225] Handling call
0905 19:52:34.798796 (+5404494us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:34.798808 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.809029 (+ 10221us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.821886 (+ 12857us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.822761 (+ 875us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5404006}
I20250905 19:52:34.821741 1171 ts_tablet_manager.cc:1397] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.112s user 0.004s sys 0.000s
I20250905 19:52:34.823187 2379 tablet_replica.cc:331] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.823356 2379 raft_consensus.cc:2241] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.823367 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8503f6918ffb44efa36214633c93e31d (table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]) successfully deleted
I20250905 19:52:34.823522 2379 raft_consensus.cc:2270] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.824055 2379 ts_tablet_manager.cc:1905] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.823892 1171 raft_consensus.cc:357] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.825276 1171 raft_consensus.cc:383] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.825361 1171 raft_consensus.cc:738] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.825501 2789 raft_consensus.cc:513] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.825555 1171 consensus_queue.cc:260] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.825824 2900 raft_consensus.cc:2802] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:34.825912 2900 raft_consensus.cc:491] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:34.825971 2900 raft_consensus.cc:3058] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.826120 1171 ts_tablet_manager.cc:1428] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:34.826761 1171 tablet_bootstrap.cc:492] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.826845 1170 tablet_bootstrap.cc:492] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.826975 1170 ts_tablet_manager.cc:1397] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:34.827595 1170 raft_consensus.cc:357] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.827740 1170 raft_consensus.cc:383] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.827795 1170 raft_consensus.cc:738] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.828007 1170 consensus_queue.cc:260] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.828510 1170 ts_tablet_manager.cc:1428] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 19:52:34.828471 2900 raft_consensus.cc:513] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.828754 1170 tablet_bootstrap.cc:492] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.829255 2900 leader_election.cc:290] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.829424 2900 raft_consensus.cc:2747] T 48be704b238e481089c727b3e1b96a2f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.830268 1170 tablet_bootstrap.cc:654] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.837915 2379 ts_tablet_manager.cc:1918] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.838042 2379 log.cc:1199] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:34.838402 2379 ts_tablet_manager.cc:1939] T fd793bce71e344f18bb177a53ab1bfc1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.838737 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 100) took 5443 ms. Trace:
I20250905 19:52:34.838827 2379 rpcz_store.cc:276] 0905 19:52:29.395131 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.395339 (+ 208us) service_pool.cc:225] Handling call
0905 19:52:34.823138 (+5427799us) ts_tablet_manager.cc:1145] Deleting tablet fd793bce71e344f18bb177a53ab1bfc1
0905 19:52:34.823149 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.832296 (+ 9147us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.836232 (+ 3936us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.838718 (+ 2486us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5427296}
I20250905 19:52:34.839183 2379 tablet_replica.cc:331] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.839366 2379 raft_consensus.cc:2241] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.839403 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet fd793bce71e344f18bb177a53ab1bfc1 (table default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 [id=1e2786d48c8d49cfa9cbca17c164b7eb]) successfully deleted
I20250905 19:52:34.839536 2379 raft_consensus.cc:2270] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.840097 2379 ts_tablet_manager.cc:1905] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.842322 2789 leader_election.cc:290] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.846905 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6fe4620dda4b1296416e01f8ffee29" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.846961 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:34.848664 1170 tablet_bootstrap.cc:492] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.848803 1170 ts_tablet_manager.cc:1397] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.002s
I20250905 19:52:34.848730 1171 tablet_bootstrap.cc:654] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.849159 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.849501 1170 raft_consensus.cc:357] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.849651 1170 raft_consensus.cc:383] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.849699 1170 raft_consensus.cc:738] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.849867 1170 consensus_queue.cc:260] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.850288 1170 ts_tablet_manager.cc:1428] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:34.850308 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f034b8cb5d8943938cb2b58c33b45494" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.850553 1170 tablet_bootstrap.cc:492] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.851038 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.851869 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:34.852188 1170 tablet_bootstrap.cc:654] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:34.852595 848 leader_election.cc:343] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.852975 848 leader_election.cc:343] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.853247 848 leader_election.cc:343] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.853489 848 leader_election.cc:343] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:34.853713 848 leader_election.cc:343] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.855331 2379 ts_tablet_manager.cc:1918] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.855435 2379 log.cc:1199] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d296428b70f9411190b6c408d81639f2
I20250905 19:52:34.855760 2379 ts_tablet_manager.cc:1939] T d296428b70f9411190b6c408d81639f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:34.856109 848 consensus_peers.cc:489] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c46ea498d6a14d1ebdffea415cfc7114. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:34.857493 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 101) took 5461 ms. Trace:
I20250905 19:52:34.857586 2379 rpcz_store.cc:276] 0905 19:52:29.395742 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.396541 (+ 799us) service_pool.cc:225] Handling call
0905 19:52:34.839135 (+5442594us) ts_tablet_manager.cc:1145] Deleting tablet d296428b70f9411190b6c408d81639f2
0905 19:52:34.839146 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.843148 (+ 4002us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.855228 (+ 12080us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.857474 (+ 2246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5441738}
I20250905 19:52:34.858093 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d296428b70f9411190b6c408d81639f2 (table default.Table_63b0eb8b21074d33a20cc13f66b38ad9 [id=80e5ee2179c744f7beccf76fc5ceb396]) successfully deleted
W20250905 19:52:34.863236 712 leader_election.cc:343] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:34.863415 712 leader_election.cc:304] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:34.864073 1849 raft_consensus.cc:3058] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.867486 1849 raft_consensus.cc:2747] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.863912 712 leader_election.cc:304] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:34.868228 1849 raft_consensus.cc:2747] T e44e88a0df444fe5b156a8fdf3199507 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.869195 1539 catalog_manager.cc:5582] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:34.870249 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:34.870867 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]). Expected version 1 got 0
I20250905 19:52:34.871783 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:34.877712 2379 tablet_replica.cc:331] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.877951 2379 raft_consensus.cc:2241] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.878118 2379 raft_consensus.cc:2270] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.878229 1849 raft_consensus.cc:491] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.878382 1849 raft_consensus.cc:513] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.878659 2379 ts_tablet_manager.cc:1905] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.872362 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:34.879200 1849 leader_election.cc:290] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.880951 1171 tablet_bootstrap.cc:492] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.881296 1171 ts_tablet_manager.cc:1397] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.001s
I20250905 19:52:34.873641 2376 log.cc:1199] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/44769f1170354870ae2d34fdb67ffbae
I20250905 19:52:34.884537 2376 ts_tablet_manager.cc:1939] T 44769f1170354870ae2d34fdb67ffbae P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:34.885438 1171 raft_consensus.cc:357] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.887377 1171 raft_consensus.cc:383] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.888444 1171 raft_consensus.cc:738] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.890728 2379 ts_tablet_manager.cc:1918] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:34.890847 2379 log.cc:1199] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6d12c7fbb84849c68274810b3fdcba04
I20250905 19:52:34.891147 2379 ts_tablet_manager.cc:1939] T 6d12c7fbb84849c68274810b3fdcba04 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.892526 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 102) took 5495 ms. Trace:
I20250905 19:52:34.832463 1169 tablet_bootstrap.cc:654] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.892684 2652 log.cc:1199] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:34.892620 2379 rpcz_store.cc:276] 0905 19:52:29.396773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.396937 (+ 164us) service_pool.cc:225] Handling call
0905 19:52:34.857841 (+5460904us) ts_tablet_manager.cc:1145] Deleting tablet 6d12c7fbb84849c68274810b3fdcba04
0905 19:52:34.877635 (+ 19794us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.886599 (+ 8964us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.890621 (+ 4022us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.892503 (+ 1882us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5460359}
I20250905 19:52:34.892947 2379 tablet_replica.cc:331] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.893163 2379 raft_consensus.cc:2241] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.893332 2379 raft_consensus.cc:2270] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.893337 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6d12c7fbb84849c68274810b3fdcba04 (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:34.893388 2652 ts_tablet_manager.cc:1939] T 5bd9380e8ffc432da4a684934c2df68f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:34.839818 984 consensus_peers.cc:489] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:34.893631 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 53861.7us
I20250905 19:52:34.893889 2379 ts_tablet_manager.cc:1905] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:34.895071 711 leader_election.cc:343] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:34.864655 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.866222 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.867056 2496 raft_consensus.cc:491] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.876216 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "421b28d047124222ae9cbc603ec22cee" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.877038 1051 raft_consensus.cc:2466] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.889498 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6fe4620dda4b1296416e01f8ffee29" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.890014 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.890245 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f034b8cb5d8943938cb2b58c33b45494" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.891324 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
W20250905 19:52:34.899268 848 consensus_peers.cc:489] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:34.908975 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:34.890796 1171 consensus_queue.cc:260] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.889875 1539 catalog_manager.cc:5582] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:34.885517 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 79) took 5526 ms. Trace:
I20250905 19:52:34.914912 1170 tablet_bootstrap.cc:492] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.915041 1170 ts_tablet_manager.cc:1397] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.065s user 0.010s sys 0.012s
I20250905 19:52:34.915776 1170 raft_consensus.cc:357] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.915930 1170 raft_consensus.cc:383] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.916006 1170 raft_consensus.cc:738] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.916205 1170 consensus_queue.cc:260] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.916569 1170 ts_tablet_manager.cc:1428] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 19:52:34.916826 1170 tablet_bootstrap.cc:492] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.915345 2376 rpcz_store.cc:276] 0905 19:52:29.359005 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.359142 (+ 137us) service_pool.cc:225] Handling call
0905 19:52:34.530436 (+5171294us) ts_tablet_manager.cc:1145] Deleting tablet 44769f1170354870ae2d34fdb67ffbae
0905 19:52:34.530448 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.778318 (+247870us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.786939 (+ 8621us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.885481 (+ 98542us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5094629}
I20250905 19:52:34.918462 1170 tablet_bootstrap.cc:654] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.919095 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 44769f1170354870ae2d34fdb67ffbae (table default.Table_885d880838c44eab9ff1220768e59373 [id=3235ea3e87b249e884b027b840d4fd77]) successfully deleted
I20250905 19:52:34.919273 769 raft_consensus.cc:3058] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.920717 1171 ts_tablet_manager.cc:1428] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.039s user 0.005s sys 0.000s
I20250905 19:52:34.921227 2376 tablet_replica.cc:331] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:34.923396 2379 ts_tablet_manager.cc:1918] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:34.923542 2379 log.cc:1199] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/43111a3512fb4fc0bdf4ee245b16695b
I20250905 19:52:34.922619 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:34.923872 2379 ts_tablet_manager.cc:1939] T 43111a3512fb4fc0bdf4ee245b16695b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.923560 2376 raft_consensus.cc:2241] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:34.924430 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 103) took 5527 ms. Trace:
I20250905 19:52:34.924527 2379 rpcz_store.cc:276] 0905 19:52:29.397203 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.398242 (+ 1039us) service_pool.cc:225] Handling call
0905 19:52:34.892897 (+5494655us) ts_tablet_manager.cc:1145] Deleting tablet 43111a3512fb4fc0bdf4ee245b16695b
0905 19:52:34.892908 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.902588 (+ 9680us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.914283 (+ 11695us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.924401 (+ 10118us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5492729}
I20250905 19:52:34.924595 1171 tablet_bootstrap.cc:492] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.925047 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 43111a3512fb4fc0bdf4ee245b16695b (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]) successfully deleted
I20250905 19:52:34.925691 2379 tablet_replica.cc:331] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.925873 2379 raft_consensus.cc:2241] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.926578 2379 raft_consensus.cc:2270] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.927459 2379 ts_tablet_manager.cc:1905] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.927598 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:34.929210 2376 raft_consensus.cc:2270] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.930462 1849 raft_consensus.cc:491] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.930593 1849 raft_consensus.cc:513] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.931434 1849 leader_election.cc:290] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:34.932514 711 leader_election.cc:343] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: f48f9f789b6e46ec871a0fd007770dfc
I20250905 19:52:34.933198 1849 raft_consensus.cc:491] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.933343 1849 raft_consensus.cc:513] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.934289 1849 leader_election.cc:290] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:34.935489 775 raft_consensus.cc:3058] T a3a50526bb604a218649b24cbf65a65e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.936304 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:34.939103 1849 raft_consensus.cc:491] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:34.939221 1849 raft_consensus.cc:513] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.940016 1849 leader_election.cc:290] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:34.941018 711 leader_election.cc:343] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 37894e39e6f2417fbfcd6d43f1ea2b37
I20250905 19:52:34.941107 2379 ts_tablet_manager.cc:1918] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:34.941205 2379 log.cc:1199] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:34.941581 2379 ts_tablet_manager.cc:1939] T 5bd9380e8ffc432da4a684934c2df68f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.941143 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:34.941874 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 104) took 5543 ms. Trace:
I20250905 19:52:34.941947 2379 rpcz_store.cc:276] 0905 19:52:29.398468 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.398620 (+ 152us) service_pool.cc:225] Handling call
0905 19:52:34.925642 (+5527022us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:34.925652 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.935490 (+ 9838us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.940998 (+ 5508us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.941856 (+ 858us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5525394}
I20250905 19:52:34.942315 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 105) took 5543 ms. Trace:
I20250905 19:52:34.942390 2379 rpcz_store.cc:276] 0905 19:52:29.398818 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.398979 (+ 161us) service_pool.cc:225] Handling call
0905 19:52:34.942211 (+5543232us) ts_tablet_manager.cc:1145] Deleting tablet 813f208881ac4782874138987a710d93
0905 19:52:34.942221 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.942302 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5541925}
I20250905 19:52:34.942469 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 5bd9380e8ffc432da4a684934c2df68f (table default.Table_a21b18f7481c4713823fadf4b164b7bf [id=16bc518342ba416b9ef21427ac1429f8]) successfully deleted
I20250905 19:52:34.942631 2379 tablet_replica.cc:331] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.942801 2379 raft_consensus.cc:2241] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:34.942900 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 813f208881ac4782874138987a710d93 because the tablet was not found. No further retry: Not found: Tablet not found: 813f208881ac4782874138987a710d93
I20250905 19:52:34.942929 2379 raft_consensus.cc:2270] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.943135 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:34.946167 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:34.947197 2379 ts_tablet_manager.cc:1905] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.893838 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 73) took 3320 ms. Trace:
I20250905 19:52:34.950644 2376 ts_tablet_manager.cc:1905] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.895632 1053 raft_consensus.cc:2466] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.895923 2496 raft_consensus.cc:513] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.896824 1036 raft_consensus.cc:3058] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.952515 2496 leader_election.cc:290] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:34.896070 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:34.952677 1049 raft_consensus.cc:3058] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.954137 1171 tablet_bootstrap.cc:654] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.954489 2379 ts_tablet_manager.cc:1918] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:34.954584 2379 log.cc:1199] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/f48f9f789b6e46ec871a0fd007770dfc
I20250905 19:52:34.954630 1036 raft_consensus.cc:2466] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.955055 1049 raft_consensus.cc:2466] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.922227 1169 tablet_bootstrap.cc:492] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:34.955416 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.955556 1169 ts_tablet_manager.cc:1397] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.161s user 0.001s sys 0.010s
I20250905 19:52:34.955557 774 raft_consensus.cc:3058] T d6b44c00d3b54af6b0752575e6898839 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.956043 2379 ts_tablet_manager.cc:1939] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:34.956406 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 106) took 5557 ms. Trace:
I20250905 19:52:34.956394 1169 raft_consensus.cc:357] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.956539 1169 raft_consensus.cc:383] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.956583 1169 raft_consensus.cc:738] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.956482 2379 rpcz_store.cc:276] 0905 19:52:29.399211 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.399375 (+ 164us) service_pool.cc:225] Handling call
0905 19:52:34.942588 (+5543213us) ts_tablet_manager.cc:1145] Deleting tablet f48f9f789b6e46ec871a0fd007770dfc
0905 19:52:34.942599 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.951512 (+ 8913us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.954372 (+ 2860us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.956388 (+ 2016us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5542271}
I20250905 19:52:34.956884 1169 consensus_queue.cc:260] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.957042 2379 tablet_replica.cc:331] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:34.957227 2379 raft_consensus.cc:2241] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.957355 2379 raft_consensus.cc:2270] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.957870 2379 ts_tablet_manager.cc:1905] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:34.940747 2924 raft_consensus.cc:491] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:34.958765 2924 raft_consensus.cc:513] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.959367 2924 leader_election.cc:290] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:34.896730 1046 raft_consensus.cc:2466] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:34.896782 1044 raft_consensus.cc:3058] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.960575 1170 tablet_bootstrap.cc:492] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:34.960727 1170 ts_tablet_manager.cc:1397] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.010s
I20250905 19:52:34.961453 1170 raft_consensus.cc:357] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.961614 1170 raft_consensus.cc:383] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.961691 1170 raft_consensus.cc:738] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.961885 1170 consensus_queue.cc:260] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.962152 1044 raft_consensus.cc:2466] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:34.962347 1170 ts_tablet_manager.cc:1428] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:34.962604 1170 tablet_bootstrap.cc:492] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:34.962706 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:34.963004 1169 ts_tablet_manager.cc:1428] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:34.963688 1169 tablet_bootstrap.cc:492] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:34.964088 986 leader_election.cc:343] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.964381 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet f48f9f789b6e46ec871a0fd007770dfc (table default.Table_2af22804e18e430ea738862d0bade226 [id=82e44ec5bc0e43e2ab11552c4127f225]) successfully deleted
I20250905 19:52:34.964834 1170 tablet_bootstrap.cc:654] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:34.964805 986 leader_election.cc:343] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: ac5c88cf8d15433a940e937f0a225582
W20250905 19:52:34.965198 984 leader_election.cc:343] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:34.965286 984 leader_election.cc:304] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.966188 2924 raft_consensus.cc:2747] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:34.966257 1171 tablet_bootstrap.cc:492] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:34.966424 1171 ts_tablet_manager.cc:1397] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.003s
W20250905 19:52:34.966415 984 leader_election.cc:343] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:34.966500 984 leader_election.cc:304] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:34.966820 2924 raft_consensus.cc:2747] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:34.934770 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:34.901495 2931 consensus_queue.cc:1035] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.048s
I20250905 19:52:34.967526 1171 raft_consensus.cc:357] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.967756 1171 raft_consensus.cc:383] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:34.967833 1171 raft_consensus.cc:738] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:34.976459 1171 consensus_queue.cc:260] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:34.977108 1171 ts_tablet_manager.cc:1428] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:34.977444 1171 tablet_bootstrap.cc:492] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:34.951534 2652 rpcz_store.cc:276] 0905 19:52:31.573048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.573350 (+ 302us) service_pool.cc:225] Handling call
0905 19:52:34.667393 (+3094043us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:34.667405 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.792870 (+125465us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.812274 (+ 19404us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.893822 (+ 81548us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3093323}
I20250905 19:52:34.978379 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 5bd9380e8ffc432da4a684934c2df68f (table default.Table_a21b18f7481c4713823fadf4b164b7bf [id=16bc518342ba416b9ef21427ac1429f8]) successfully deleted
I20250905 19:52:34.982995 1169 tablet_bootstrap.cc:654] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.983937 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52c99a76802b481c98f77dbd90de7cbb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.984140 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "0d93a328d4934e5294ec66f739eb240b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.984220 909 raft_consensus.cc:2466] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:34.984566 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f48f9f789b6e46ec871a0fd007770dfc" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:34.984664 2652 tablet_replica.cc:331] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:34.984889 2652 raft_consensus.cc:2241] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:34.985034 2652 raft_consensus.cc:2270] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:34.986173 771 raft_consensus.cc:3058] T f2d929ec75d34cddad5286d8e37fcefb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:34.992046 1171 tablet_bootstrap.cc:654] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:34.997117 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:35.000378 2379 ts_tablet_manager.cc:1918] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.000531 2379 log.cc:1199] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a8a32f742eaf473e9f6f9bf214f53679
I20250905 19:52:35.000905 2379 ts_tablet_manager.cc:1939] T a8a32f742eaf473e9f6f9bf214f53679 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.001250 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 107) took 5601 ms. Trace:
I20250905 19:52:35.001448 2652 ts_tablet_manager.cc:1905] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.001339 2379 rpcz_store.cc:276] 0905 19:52:29.399610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.399821 (+ 211us) service_pool.cc:225] Handling call
0905 19:52:34.956992 (+5557171us) ts_tablet_manager.cc:1145] Deleting tablet a8a32f742eaf473e9f6f9bf214f53679
0905 19:52:34.957002 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:34.986436 (+ 29434us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:34.989425 (+ 2989us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.001230 (+ 11805us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5556644}
I20250905 19:52:35.001709 2379 tablet_replica.cc:331] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.001902 2379 raft_consensus.cc:2241] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.001932 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a8a32f742eaf473e9f6f9bf214f53679 (table default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]) successfully deleted
I20250905 19:52:35.002041 2379 raft_consensus.cc:2270] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.007706 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "37894e39e6f2417fbfcd6d43f1ea2b37" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:35.009265 849 consensus_peers.cc:489] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.010039 2838 consensus_queue.cc:1035] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:35.015532 2379 ts_tablet_manager.cc:1905] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.018735 2422 raft_consensus.cc:491] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.021238 2422 raft_consensus.cc:513] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.021993 2422 leader_election.cc:290] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.022768 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7155cf5abba4e2cbdce583406f94fad" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.025053 848 leader_election.cc:343] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.023267 1539 catalog_manager.cc:5582] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.032655 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:35.035449 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:35.035908 1171 tablet_bootstrap.cc:492] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.037909 1169 tablet_bootstrap.cc:492] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:35.038250 1169 ts_tablet_manager.cc:1397] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.075s user 0.003s sys 0.003s
I20250905 19:52:35.038000 1171 ts_tablet_manager.cc:1397] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.060s user 0.001s sys 0.008s
I20250905 19:52:35.038763 2379 ts_tablet_manager.cc:1918] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.038878 2379 log.cc:1199] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/24bcd21f19ba465aa550102e0fccb674
I20250905 19:52:35.039466 2379 ts_tablet_manager.cc:1939] T 24bcd21f19ba465aa550102e0fccb674 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.039814 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 108) took 5639 ms. Trace:
I20250905 19:52:35.039907 2379 rpcz_store.cc:276] 0905 19:52:29.400076 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.400950 (+ 874us) service_pool.cc:225] Handling call
0905 19:52:35.001660 (+5600710us) ts_tablet_manager.cc:1145] Deleting tablet 24bcd21f19ba465aa550102e0fccb674
0905 19:52:35.001672 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.032151 (+ 30479us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.038652 (+ 6501us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.039794 (+ 1142us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5599641}
I20250905 19:52:35.040293 2379 tablet_replica.cc:331] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.040479 2379 raft_consensus.cc:2241] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.040540 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 24bcd21f19ba465aa550102e0fccb674 (table default.Table_65d1d80076a24f24a1c5284f9e90ec10 [id=c7b558dcabda400d9ba6e1850e0a61eb]) successfully deleted
I20250905 19:52:35.040619 2379 raft_consensus.cc:2270] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.040300 1169 raft_consensus.cc:357] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.040726 1169 raft_consensus.cc:383] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.040906 1169 raft_consensus.cc:738] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.041366 2379 ts_tablet_manager.cc:1905] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.041630 1169 consensus_queue.cc:260] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.042522 1170 tablet_bootstrap.cc:492] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.042645 1170 ts_tablet_manager.cc:1397] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.080s user 0.008s sys 0.008s
I20250905 19:52:35.043393 1170 raft_consensus.cc:357] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.043548 1170 raft_consensus.cc:383] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.043596 1170 raft_consensus.cc:738] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.043787 1170 consensus_queue.cc:260] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.045277 1169 ts_tablet_manager.cc:1428] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250905 19:52:35.048532 2931 raft_consensus.cc:491] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:35.048691 2931 raft_consensus.cc:513] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.049047 2838 consensus_queue.cc:1035] T 8e0c478efe044ad0867e47f49374a66b P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:35.049294 2931 leader_election.cc:290] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.049950 2931 raft_consensus.cc:491] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:35.050071 2931 raft_consensus.cc:513] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.050602 2931 leader_election.cc:290] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.050716 1171 raft_consensus.cc:357] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:35.051283 986 leader_election.cc:343] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 54628965b82e470d811eff95ffc56c5f
W20250905 19:52:35.051648 986 leader_election.cc:343] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 723168b075874dfcb3eb50c5134c499e
W20250905 19:52:35.051644 984 leader_election.cc:343] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 54628965b82e470d811eff95ffc56c5f
I20250905 19:52:35.051762 984 leader_election.cc:304] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.052182 2931 raft_consensus.cc:2747] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:35.051991 1171 raft_consensus.cc:383] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250905 19:52:35.053193 984 leader_election.cc:343] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 723168b075874dfcb3eb50c5134c499e
I20250905 19:52:35.053287 984 leader_election.cc:304] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.053700 2931 raft_consensus.cc:2747] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:52:35.053992 848 consensus_peers.cc:489] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8503f6918ffb44efa36214633c93e31d. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:35.054368 849 leader_election.cc:343] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.054576 849 leader_election.cc:304] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.054644 1169 tablet_bootstrap.cc:492] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:35.055554 1170 ts_tablet_manager.cc:1428] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:35.055927 1170 tablet_bootstrap.cc:492] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.052870 1539 catalog_manager.cc:5582] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.057767 1170 tablet_bootstrap.cc:654] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.060214 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:35.053256 1171 raft_consensus.cc:738] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.061635 2838 raft_consensus.cc:2747] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.060984 2376 ts_tablet_manager.cc:1918] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.061803 1171 consensus_queue.cc:260] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.067836 1171 ts_tablet_manager.cc:1428] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.029s user 0.001s sys 0.002s
I20250905 19:52:35.062978 2376 log.cc:1199] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/c46ea498d6a14d1ebdffea415cfc7114
I20250905 19:52:35.068506 1171 tablet_bootstrap.cc:492] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:35.068430 849 leader_election.cc:343] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.068814 849 leader_election.cc:304] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.069124 2838 raft_consensus.cc:2747] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.072031 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:35.076160 2376 ts_tablet_manager.cc:1939] T c46ea498d6a14d1ebdffea415cfc7114 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:35.077350 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet d495b6ba92e84b588b4780c2896195c0 consensus state has no leader
I20250905 19:52:35.078567 2379 ts_tablet_manager.cc:1918] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.078696 2379 log.cc:1199] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6221d7d058944bdb9aa730791cf2417b
I20250905 19:52:35.078997 2379 ts_tablet_manager.cc:1939] T 6221d7d058944bdb9aa730791cf2417b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.079556 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 3 got 0
W20250905 19:52:35.081112 849 leader_election.cc:343] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.081311 1170 tablet_bootstrap.cc:492] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.081475 1170 ts_tablet_manager.cc:1397] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.002s
I20250905 19:52:35.081496 849 leader_election.cc:304] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.082140 2838 raft_consensus.cc:2747] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.082259 1170 raft_consensus.cc:357] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.082465 1170 raft_consensus.cc:383] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.082568 1170 raft_consensus.cc:738] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.082809 1170 consensus_queue.cc:260] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.083330 1170 ts_tablet_manager.cc:1428] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.083636 1170 tablet_bootstrap.cc:492] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.083171 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 80) took 5645 ms. Trace:
I20250905 19:52:35.085942 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 110) took 5512 ms. Trace:
I20250905 19:52:35.086072 2379 rpcz_store.cc:276] 0905 19:52:29.573960 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.574026 (+ 66us) service_pool.cc:225] Handling call
0905 19:52:35.040244 (+5466218us) ts_tablet_manager.cc:1145] Deleting tablet 6221d7d058944bdb9aa730791cf2417b
0905 19:52:35.040254 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.067441 (+ 27187us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.078446 (+ 11005us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.085920 (+ 7474us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5465969}
I20250905 19:52:35.086436 2379 tablet_replica.cc:331] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.086676 2379 raft_consensus.cc:2241] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.086709 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]) successfully deleted
I20250905 19:52:35.086851 2379 raft_consensus.cc:2270] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.084865 2376 rpcz_store.cc:276] 0905 19:52:29.437393 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.437483 (+ 90us) service_pool.cc:225] Handling call
0905 19:52:34.920285 (+5482802us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:34.920298 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.010143 (+ 89845us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.058980 (+ 48837us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.082398 (+ 23418us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5481427}
I20250905 19:52:35.087374 2379 ts_tablet_manager.cc:1905] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.087512 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet c46ea498d6a14d1ebdffea415cfc7114 (table default.Table_f542b700d5364bfeb6e1e0111834fab0 [id=27f7071a498d467aa9047b07dac76b64]) successfully deleted
I20250905 19:52:35.084412 849 leader_election.cc:304] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.088500 1170 tablet_bootstrap.cc:654] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.084678 1539 catalog_manager.cc:5582] T 0d93a328d4934e5294ec66f739eb240b P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.088800 2838 raft_consensus.cc:2802] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.088647 1169 tablet_bootstrap.cc:654] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.089275 2838 raft_consensus.cc:695] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:35.089624 2838 consensus_queue.cc:237] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.089547 2376 tablet_replica.cc:331] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:35.092060 1849 raft_consensus.cc:491] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.092227 1849 raft_consensus.cc:513] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.092821 1849 leader_election.cc:290] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.092788 2376 raft_consensus.cc:2241] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.093456 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:35.097473 2376 raft_consensus.cc:2270] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.098308 2379 ts_tablet_manager.cc:1918] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.098941 2496 raft_consensus.cc:491] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.099079 2496 raft_consensus.cc:513] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.099717 2496 leader_election.cc:290] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.100750 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd1280fefc3e43db9928567a734644f1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.099787 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
W20250905 19:52:35.101253 986 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.101559 1170 tablet_bootstrap.cc:492] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.101764 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd1280fefc3e43db9928567a734644f1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.098466 2379 log.cc:1199] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ba161282480e4736889559a2e37b195f
I20250905 19:52:35.101855 1170 ts_tablet_manager.cc:1397] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.003s
W20250905 19:52:35.102348 984 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.102453 984 leader_election.cc:304] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.102695 1170 raft_consensus.cc:357] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.102833 2496 raft_consensus.cc:2747] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.102882 1170 raft_consensus.cc:383] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.102963 1170 raft_consensus.cc:738] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.103333 1170 consensus_queue.cc:260] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.104022 1170 ts_tablet_manager.cc:1428] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:35.104339 1170 tablet_bootstrap.cc:492] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.105365 2376 ts_tablet_manager.cc:1905] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.105777 1171 tablet_bootstrap.cc:654] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.106549 1170 tablet_bootstrap.cc:654] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.107705 2379 ts_tablet_manager.cc:1939] T ba161282480e4736889559a2e37b195f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.108552 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 95) took 5722 ms. Trace:
I20250905 19:52:35.108668 2379 rpcz_store.cc:276] 0905 19:52:29.385613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.385653 (+ 40us) service_pool.cc:225] Handling call
0905 19:52:35.086361 (+5700708us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:35.086371 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.093652 (+ 7281us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.098183 (+ 4531us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.108495 (+ 10312us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5509989}
I20250905 19:52:35.109035 2379 tablet_replica.cc:331] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.109243 2379 raft_consensus.cc:2241] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.109330 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ba161282480e4736889559a2e37b195f (table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]) successfully deleted
I20250905 19:52:35.109424 2379 raft_consensus.cc:2270] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.110141 2379 ts_tablet_manager.cc:1905] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.107336 1539 catalog_manager.cc:5582] T d6b44c00d3b54af6b0752575e6898839 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.111059 849 leader_election.cc:304] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.111362 2413 raft_consensus.cc:2802] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.111514 2413 raft_consensus.cc:491] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.111665 2413 raft_consensus.cc:3058] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.114619 2413 raft_consensus.cc:513] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.115427 2413 leader_election.cc:290] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.114897 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]). Expected version 1 got 0
I20250905 19:52:35.116429 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:35.117105 848 leader_election.cc:343] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.118618 1170 tablet_bootstrap.cc:492] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.118779 1170 ts_tablet_manager.cc:1397] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.002s
I20250905 19:52:35.119570 1170 raft_consensus.cc:357] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.119747 1170 raft_consensus.cc:383] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.119848 1170 raft_consensus.cc:738] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.120102 1170 consensus_queue.cc:260] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.120481 2652 ts_tablet_manager.cc:1918] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.120757 1170 ts_tablet_manager.cc:1428] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:35.121050 1170 tablet_bootstrap.cc:492] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.121285 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]). Expected version 1 got 0
I20250905 19:52:35.123575 1170 tablet_bootstrap.cc:654] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.124557 2379 ts_tablet_manager.cc:1918] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.124691 2379 log.cc:1199] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9193487b4ab74dafa76a386ddcaf674b
I20250905 19:52:35.125048 2379 ts_tablet_manager.cc:1939] T 9193487b4ab74dafa76a386ddcaf674b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.125211 769 raft_consensus.cc:3058] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.130463 849 leader_election.cc:304] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.130865 2738 raft_consensus.cc:2802] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.130856 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 111) took 5545 ms. Trace:
I20250905 19:52:35.131057 2738 raft_consensus.cc:695] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:35.130981 2379 rpcz_store.cc:276] 0905 19:52:29.585375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.596266 (+ 10891us) service_pool.cc:225] Handling call
0905 19:52:35.108951 (+5512685us) ts_tablet_manager.cc:1145] Deleting tablet 9193487b4ab74dafa76a386ddcaf674b
0905 19:52:35.108960 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.113273 (+ 4313us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.124388 (+ 11115us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.130817 (+ 6429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5512483}
I20250905 19:52:35.131340 2379 tablet_replica.cc:331] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.131539 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9193487b4ab74dafa76a386ddcaf674b (table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]) successfully deleted
I20250905 19:52:35.131567 2379 raft_consensus.cc:2241] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.131736 2379 raft_consensus.cc:2270] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.132119 2738 consensus_queue.cc:237] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.132465 2379 ts_tablet_manager.cc:1905] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.134557 1849 raft_consensus.cc:491] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.134707 1849 raft_consensus.cc:513] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.135358 1849 leader_election.cc:290] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.135844 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]). Expected version 1 got 0
I20250905 19:52:35.137188 2413 raft_consensus.cc:491] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.137434 2413 raft_consensus.cc:513] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.137775 1171 tablet_bootstrap.cc:492] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.137939 1171 ts_tablet_manager.cc:1397] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.070s user 0.000s sys 0.005s
I20250905 19:52:35.138410 2413 leader_election.cc:290] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.138880 1171 raft_consensus.cc:357] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.139068 1171 raft_consensus.cc:383] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.139143 1171 raft_consensus.cc:738] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.139375 1171 consensus_queue.cc:260] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:35.139612 848 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 24bcd21f19ba465aa550102e0fccb674
I20250905 19:52:35.139890 1171 ts_tablet_manager.cc:1428] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:35.140228 1171 tablet_bootstrap.cc:492] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.142062 1171 tablet_bootstrap.cc:654] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.141561 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:35.151707 849 leader_election.cc:304] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.152297 849 leader_election.cc:304] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.152745 849 leader_election.cc:304] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.152997 2422 raft_consensus.cc:2802] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.153117 1171 tablet_bootstrap.cc:492] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.153748 1171 ts_tablet_manager.cc:1397] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.002s
I20250905 19:52:35.157179 2903 raft_consensus.cc:2802] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.157284 2903 raft_consensus.cc:491] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.157379 2903 raft_consensus.cc:3058] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.157449 1170 tablet_bootstrap.cc:492] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.157611 1170 ts_tablet_manager.cc:1397] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.004s
I20250905 19:52:35.144095 1539 catalog_manager.cc:5582] T 7f34bdf651d1412690039dc6ea1095d0 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.158329 1170 raft_consensus.cc:357] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.161629 1170 raft_consensus.cc:383] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.161717 1170 raft_consensus.cc:738] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.161953 1170 consensus_queue.cc:260] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.160945 1539 catalog_manager.cc:5582] T c4b31443b0f840749e12435e197a204a P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.162436 1170 ts_tablet_manager.cc:1428] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:35.164477 2903 raft_consensus.cc:513] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.165133 2903 leader_election.cc:290] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.165307 2903 raft_consensus.cc:2802] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.165529 2903 raft_consensus.cc:695] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:35.165791 2903 consensus_queue.cc:237] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.167588 1170 tablet_bootstrap.cc:492] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.170121 1170 tablet_bootstrap.cc:654] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.170882 2422 raft_consensus.cc:695] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:35.171250 1171 raft_consensus.cc:357] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.171443 1171 raft_consensus.cc:383] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.171540 1171 raft_consensus.cc:738] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.171283 2422 consensus_queue.cc:237] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.171761 1171 consensus_queue.cc:260] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.172328 1171 ts_tablet_manager.cc:1428] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:35.172664 1171 tablet_bootstrap.cc:492] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.174461 1171 tablet_bootstrap.cc:654] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.175695 2379 ts_tablet_manager.cc:1918] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.177619 2413 raft_consensus.cc:491] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.177845 2413 raft_consensus.cc:513] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.174851 1539 catalog_manager.cc:5582] T db098a883a61474fb06cb166041ee3e4 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:35.179162 2413 leader_election.cc:290] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.179829 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.180429 2379 log.cc:1199] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9db56577a26e407b97fc33158c2c466c
I20250905 19:52:35.180876 2379 ts_tablet_manager.cc:1939] T 9db56577a26e407b97fc33158c2c466c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.181694 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 121) took 3693 ms. Trace:
I20250905 19:52:35.181802 2379 rpcz_store.cc:276] 0905 19:52:31.488451 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.488499 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:35.131271 (+3642772us) ts_tablet_manager.cc:1145] Deleting tablet 9db56577a26e407b97fc33158c2c466c
0905 19:52:35.131279 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.135478 (+ 4199us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.175539 (+ 40061us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.181610 (+ 6071us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3642511}
I20250905 19:52:35.182238 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 122) took 3606 ms. Trace:
I20250905 19:52:35.182325 2379 rpcz_store.cc:276] 0905 19:52:31.575807 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.575879 (+ 72us) service_pool.cc:225] Handling call
0905 19:52:35.182076 (+3606197us) ts_tablet_manager.cc:1145] Deleting tablet 3c824f5591044c4198a51a25661f71e5
0905 19:52:35.182086 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.182178 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605947}
I20250905 19:52:35.182523 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9db56577a26e407b97fc33158c2c466c (table default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]) successfully deleted
I20250905 19:52:35.182626 2379 tablet_replica.cc:331] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.182873 2379 raft_consensus.cc:2241] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.183059 2379 raft_consensus.cc:2270] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:35.183202 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
I20250905 19:52:35.183629 2379 ts_tablet_manager.cc:1905] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.183430 1539 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:35.186210 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:35.180570 848 leader_election.cc:343] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.183221 2376 ts_tablet_manager.cc:1918] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.186997 2376 log.cc:1199] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8503f6918ffb44efa36214633c93e31d
W20250905 19:52:35.187161 848 leader_election.cc:343] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.189682 2376 ts_tablet_manager.cc:1939] T 8503f6918ffb44efa36214633c93e31d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:35.192438 712 leader_election.cc:304] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:35.192906 1849 raft_consensus.cc:2747] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.192926 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 81) took 5745 ms. Trace:
I20250905 19:52:35.193486 712 leader_election.cc:304] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:35.193902 1849 raft_consensus.cc:2802] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.193995 1849 raft_consensus.cc:491] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.194104 1849 raft_consensus.cc:3058] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.194718 2376 rpcz_store.cc:276] 0905 19:52:29.446761 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.446852 (+ 91us) service_pool.cc:225] Handling call
0905 19:52:35.088414 (+5641562us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:35.088427 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.143367 (+ 54940us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.181834 (+ 38467us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.192509 (+ 10675us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5640210}
I20250905 19:52:35.196996 1849 raft_consensus.cc:513] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.197711 1849 leader_election.cc:290] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.198182 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8503f6918ffb44efa36214633c93e31d (table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]) successfully deleted
W20250905 19:52:35.199429 711 leader_election.cc:343] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:35.199081 2376 tablet_replica.cc:331] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:35.200721 712 leader_election.cc:304] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:35.200984 712 raft_consensus.cc:2684] T f48f9f789b6e46ec871a0fd007770dfc P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:35.202643 712 leader_election.cc:304] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:35.203136 1849 raft_consensus.cc:2747] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.128494 1169 tablet_bootstrap.cc:492] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:35.136255 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:35.203516 1169 ts_tablet_manager.cc:1397] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.151s user 0.007s sys 0.001s
I20250905 19:52:35.204011 2652 log.cc:1199] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:35.204442 2652 ts_tablet_manager.cc:1939] T 8d38452d3a0641ffa4af5d935f8ad9f0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:35.204398 1169 raft_consensus.cc:357] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.204571 1169 raft_consensus.cc:383] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.204630 1169 raft_consensus.cc:738] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.204856 1169 consensus_queue.cc:260] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.205046 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 76) took 3629 ms. Trace:
I20250905 19:52:35.205157 2652 rpcz_store.cc:276] 0905 19:52:31.575912 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.576065 (+ 153us) service_pool.cc:225] Handling call
0905 19:52:34.984599 (+3408534us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:34.984621 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.084137 (+ 99516us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.120365 (+ 36228us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.204968 (+ 84603us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3408342}
I20250905 19:52:35.205444 2379 ts_tablet_manager.cc:1918] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.205329 1169 ts_tablet_manager.cc:1428] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.205581 2379 log.cc:1199] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/1f1a8e7059b348cba34069ef49bf1aad
I20250905 19:52:35.205724 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8d38452d3a0641ffa4af5d935f8ad9f0 (table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]) successfully deleted
I20250905 19:52:35.205822 1169 tablet_bootstrap.cc:492] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:35.206156 2652 tablet_replica.cc:331] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:35.206399 2652 raft_consensus.cc:2241] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.175654 2924 raft_consensus.cc:491] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.206661 2924 raft_consensus.cc:513] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.206995 2379 ts_tablet_manager.cc:1939] T 1f1a8e7059b348cba34069ef49bf1aad P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.207442 1169 tablet_bootstrap.cc:654] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.207818 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.208074 1171 tablet_bootstrap.cc:492] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:35.208374 984 leader_election.cc:343] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.208951 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 123) took 3632 ms. Trace:
I20250905 19:52:35.209164 2924 leader_election.cc:290] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.209229 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.210228 2379 rpcz_store.cc:276] 0905 19:52:31.576773 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.576826 (+ 53us) service_pool.cc:225] Handling call
0905 19:52:35.182535 (+3605709us) ts_tablet_manager.cc:1145] Deleting tablet 1f1a8e7059b348cba34069ef49bf1aad
0905 19:52:35.182543 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.200456 (+ 17913us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.205293 (+ 4837us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.208917 (+ 3624us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3605520}
I20250905 19:52:35.210914 2379 tablet_replica.cc:331] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.211057 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 1f1a8e7059b348cba34069ef49bf1aad (table default.Table_7186efc2caef4940ade686830c963ba1 [id=0a06a88771a94e60b5bb57fa7f7d8215]) successfully deleted
I20250905 19:52:35.211411 2379 raft_consensus.cc:2241] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.211580 2379 raft_consensus.cc:2270] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.212157 2379 ts_tablet_manager.cc:1905] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.215376 906 raft_consensus.cc:2466] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.215715 1171 ts_tablet_manager.cc:1397] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.036s user 0.002s sys 0.007s
I20250905 19:52:35.215998 986 leader_election.cc:304] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.216290 2930 raft_consensus.cc:2802] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.216356 2930 raft_consensus.cc:491] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.216424 2930 raft_consensus.cc:3058] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.218956 2930 raft_consensus.cc:513] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.219537 2930 leader_election.cc:290] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.216667 1171 raft_consensus.cc:357] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.220046 1171 raft_consensus.cc:383] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.220125 1171 raft_consensus.cc:738] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.220373 1171 consensus_queue.cc:260] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.220808 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.216009 2376 raft_consensus.cc:2241] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.221001 906 raft_consensus.cc:3058] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.224037 906 raft_consensus.cc:2466] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.224345 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:35.224450 1171 ts_tablet_manager.cc:1428] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250905 19:52:35.224807 1171 tablet_bootstrap.cc:492] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.224529 2376 raft_consensus.cc:2270] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.226385 986 leader_election.cc:304] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:35.226701 2930 raft_consensus.cc:2802] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.226907 2930 raft_consensus.cc:695] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:35.227263 1171 tablet_bootstrap.cc:654] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.227373 2930 consensus_queue.cc:237] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.235888 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:35.237349 984 leader_election.cc:343] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.238555 2379 ts_tablet_manager.cc:1918] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.238716 2379 log.cc:1199] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/578a6b9224cc4d8585d6c4b241762e1b
I20250905 19:52:35.239058 2379 ts_tablet_manager.cc:1939] T 578a6b9224cc4d8585d6c4b241762e1b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.239837 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 124) took 3662 ms. Trace:
I20250905 19:52:35.242231 1171 tablet_bootstrap.cc:492] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.242386 1171 ts_tablet_manager.cc:1397] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20250905 19:52:35.243248 1171 raft_consensus.cc:357] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.243443 1171 raft_consensus.cc:383] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.243530 1171 raft_consensus.cc:738] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.243736 1171 consensus_queue.cc:260] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.244768 1171 ts_tablet_manager.cc:1428] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.245415 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7155cf5abba4e2cbdce583406f94fad" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:35.245630 1171 tablet_bootstrap.cc:492] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.247422 1171 tablet_bootstrap.cc:654] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:35.248898 848 consensus_peers.cc:489] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.250802 1170 tablet_bootstrap.cc:492] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.250962 1170 ts_tablet_manager.cc:1397] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.083s user 0.009s sys 0.012s
I20250905 19:52:35.251695 1170 raft_consensus.cc:357] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.251896 1170 raft_consensus.cc:383] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.252044 1170 raft_consensus.cc:738] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.252286 1170 consensus_queue.cc:260] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.252806 1170 ts_tablet_manager.cc:1428] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.255064 2422 raft_consensus.cc:491] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.254818 2376 ts_tablet_manager.cc:1905] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.256101 2422 raft_consensus.cc:513] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.256945 1170 tablet_bootstrap.cc:492] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.258330 1170 tablet_bootstrap.cc:654] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.259492 2422 leader_election.cc:290] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.260208 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ebab2e3698f4f60a770bde9cc019600" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.261180 848 leader_election.cc:343] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.272711 2652 raft_consensus.cc:2270] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.273556 2652 ts_tablet_manager.cc:1905] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.284765 769 raft_consensus.cc:3058] T 4d6f2e44855d4312b1c0367145e3cee8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.300235 2379 rpcz_store.cc:276] 0905 19:52:31.577676 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.577721 (+ 45us) service_pool.cc:225] Handling call
0905 19:52:35.210840 (+3633119us) ts_tablet_manager.cc:1145] Deleting tablet 578a6b9224cc4d8585d6c4b241762e1b
0905 19:52:35.210851 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.215270 (+ 4419us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.238426 (+ 23156us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.239774 (+ 1348us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3632957}
I20250905 19:52:35.300730 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 125) took 3722 ms. Trace:
I20250905 19:52:35.300822 2379 rpcz_store.cc:276] 0905 19:52:31.578337 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.578382 (+ 45us) service_pool.cc:225] Handling call
0905 19:52:35.300578 (+3722196us) ts_tablet_manager.cc:1145] Deleting tablet 3609de83732d4ad8857372acfbb5c32a
0905 19:52:35.300589 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.300708 (+ 119us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3722027}
I20250905 19:52:35.301455 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 578a6b9224cc4d8585d6c4b241762e1b (table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]) successfully deleted
W20250905 19:52:35.302062 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3609de83732d4ad8857372acfbb5c32a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3609de83732d4ad8857372acfbb5c32a already in progress: creating tablet
I20250905 19:52:35.301832 1169 tablet_bootstrap.cc:492] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:35.302452 1171 tablet_bootstrap.cc:492] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.304484 1169 ts_tablet_manager.cc:1397] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.098s user 0.006s sys 0.008s
I20250905 19:52:35.305127 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:35.305320 1046 raft_consensus.cc:3058] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.307010 1170 tablet_bootstrap.cc:492] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.307158 1170 ts_tablet_manager.cc:1397] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.050s user 0.001s sys 0.012s
I20250905 19:52:35.301139 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 126) took 3721 ms. Trace:
I20250905 19:52:35.307703 2379 rpcz_store.cc:276] 0905 19:52:31.579373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.579422 (+ 49us) service_pool.cc:225] Handling call
0905 19:52:35.301038 (+3721616us) ts_tablet_manager.cc:1145] Deleting tablet 4c76077e4adf45a7adc736b0e69032eb
0905 19:52:35.301046 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.301100 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3721468}
I20250905 19:52:35.308157 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 127) took 3728 ms. Trace:
I20250905 19:52:35.308249 2379 rpcz_store.cc:276] 0905 19:52:31.579959 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.580030 (+ 71us) service_pool.cc:225] Handling call
0905 19:52:35.308040 (+3728010us) ts_tablet_manager.cc:1145] Deleting tablet 74a7e7d9874e4d26bda3ab24a6e24adc
0905 19:52:35.308050 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.308134 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3727837}
W20250905 19:52:35.308578 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4c76077e4adf45a7adc736b0e69032eb because the tablet was not found. No further retry: Not found: Tablet not found: 4c76077e4adf45a7adc736b0e69032eb
I20250905 19:52:35.308605 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 128) took 3727 ms. Trace:
I20250905 19:52:35.308696 2379 rpcz_store.cc:276] 0905 19:52:31.580753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.582031 (+ 1278us) service_pool.cc:225] Handling call
0905 19:52:35.308519 (+3726488us) ts_tablet_manager.cc:1145] Deleting tablet 4fe94d1254d04949aa25346d8e323c3b
0905 19:52:35.308530 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.308590 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3726299}
I20250905 19:52:35.309015 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 129) took 3726 ms. Trace:
I20250905 19:52:35.309095 2379 rpcz_store.cc:276] 0905 19:52:31.582868 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.582937 (+ 69us) service_pool.cc:225] Handling call
0905 19:52:35.308907 (+3725970us) ts_tablet_manager.cc:1145] Deleting tablet a2ff206366ad4344b4df5fc6a5584f8a
0905 19:52:35.308915 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.308999 (+ 84us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3725790}
W20250905 19:52:35.309314 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 74a7e7d9874e4d26bda3ab24a6e24adc because the tablet was not found. No further retry: Not found: Tablet not found: 74a7e7d9874e4d26bda3ab24a6e24adc
I20250905 19:52:35.309365 2379 tablet_replica.cc:331] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:35.309566 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4fe94d1254d04949aa25346d8e323c3b because the tablet was not found. No further retry: Not found: Tablet not found: 4fe94d1254d04949aa25346d8e323c3b
I20250905 19:52:35.309594 2379 raft_consensus.cc:2241] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.309767 1171 ts_tablet_manager.cc:1397] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.011s
W20250905 19:52:35.310067 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a2ff206366ad4344b4df5fc6a5584f8a because tablet deleting was already in progress. No further retry: Already present: State transition of tablet a2ff206366ad4344b4df5fc6a5584f8a already in progress: creating tablet
I20250905 19:52:35.310403 1046 raft_consensus.cc:2466] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:35.310801 1171 raft_consensus.cc:357] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.311056 1171 raft_consensus.cc:383] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.311142 1171 raft_consensus.cc:738] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.311407 1171 consensus_queue.cc:260] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.311932 1171 ts_tablet_manager.cc:1428] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.312140 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:35.312356 1047 raft_consensus.cc:3058] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.317057 1169 raft_consensus.cc:357] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.317471 1169 raft_consensus.cc:383] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.320521 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:35.320984 1169 raft_consensus.cc:738] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.326876 1171 tablet_bootstrap.cc:492] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.328933 1171 tablet_bootstrap.cc:654] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.334275 1169 consensus_queue.cc:260] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.334957 1169 ts_tablet_manager.cc:1428] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.029s user 0.003s sys 0.000s
I20250905 19:52:35.335289 1169 tablet_bootstrap.cc:492] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:35.337122 2376 ts_tablet_manager.cc:1918] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.337476 2376 log.cc:1199] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d296428b70f9411190b6c408d81639f2
I20250905 19:52:35.338486 2376 ts_tablet_manager.cc:1939] T d296428b70f9411190b6c408d81639f2 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:35.339234 848 consensus_peers.cc:489] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5bd9380e8ffc432da4a684934c2df68f. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.340269 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "040e65fdb1104b07b953044f9b6b538a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.340673 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 82) took 5892 ms. Trace:
I20250905 19:52:35.341013 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5f9e807244c4e0eb1f6dd1fc2acb9c1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.340957 2376 rpcz_store.cc:276] 0905 19:52:29.448259 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.448376 (+ 117us) service_pool.cc:225] Handling call
0905 19:52:35.198489 (+5750113us) ts_tablet_manager.cc:1145] Deleting tablet d296428b70f9411190b6c408d81639f2
0905 19:52:35.198503 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.322561 (+124058us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.336827 (+ 14266us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.340526 (+ 3699us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":69,"tablet-delete.queue_time_us":5749586}
I20250905 19:52:35.341771 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "52c99a76802b481c98f77dbd90de7cbb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.341958 907 raft_consensus.cc:3058] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.342608 2376 tablet_replica.cc:331] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:35.343168 2376 raft_consensus.cc:2241] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.343334 1171 tablet_bootstrap.cc:492] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.343497 1171 ts_tablet_manager.cc:1397] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.000s
I20250905 19:52:35.343958 2376 raft_consensus.cc:2270] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.344257 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d296428b70f9411190b6c408d81639f2 (table default.Table_63b0eb8b21074d33a20cc13f66b38ad9 [id=80e5ee2179c744f7beccf76fc5ceb396]) successfully deleted
I20250905 19:52:35.344691 1171 raft_consensus.cc:357] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.345475 1171 raft_consensus.cc:383] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.345556 907 raft_consensus.cc:2466] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:35.345602 1171 raft_consensus.cc:738] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.345834 2376 ts_tablet_manager.cc:1905] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.345834 1171 consensus_queue.cc:260] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.346657 1171 ts_tablet_manager.cc:1428] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20250905 19:52:35.346858 2930 raft_consensus.cc:491] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:35.347008 2930 raft_consensus.cc:513] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.347090 1171 tablet_bootstrap.cc:492] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.348773 712 leader_election.cc:304] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:35.349133 1171 tablet_bootstrap.cc:654] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.352773 2930 leader_election.cc:290] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:35.354269 984 leader_election.cc:343] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: a3805127dd78489e8545da7414ead904
I20250905 19:52:35.356092 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ebab2e3698f4f60a770bde9cc019600" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:35.357985 986 leader_election.cc:343] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: a3805127dd78489e8545da7414ead904
I20250905 19:52:35.358100 986 leader_election.cc:304] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.358565 2930 raft_consensus.cc:2747] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:35.360883 1171 tablet_bootstrap.cc:492] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.361079 1171 ts_tablet_manager.cc:1397] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250905 19:52:35.362006 1171 raft_consensus.cc:357] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.362241 1171 raft_consensus.cc:383] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.362326 1171 raft_consensus.cc:738] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.362567 1171 consensus_queue.cc:260] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.363140 1171 ts_tablet_manager.cc:1428] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.363485 1171 tablet_bootstrap.cc:492] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.365515 1171 tablet_bootstrap.cc:654] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.366137 2789 consensus_queue.cc:1035] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:35.370085 849 leader_election.cc:343] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.370347 849 leader_election.cc:304] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.370671 2738 raft_consensus.cc:2747] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.374991 849 leader_election.cc:304] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.375663 2789 raft_consensus.cc:2802] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.375823 2789 raft_consensus.cc:695] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:35.376675 2789 consensus_queue.cc:237] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.384302 2379 raft_consensus.cc:2270] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.384568 712 leader_election.cc:304] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:35.384958 2379 ts_tablet_manager.cc:1905] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.385037 2888 raft_consensus.cc:2747] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.385183 2888 raft_consensus.cc:2747] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.384976 712 leader_election.cc:304] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:35.385823 1849 raft_consensus.cc:2802] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.385929 1849 raft_consensus.cc:695] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:35.386153 1849 consensus_queue.cc:237] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.389070 1170 raft_consensus.cc:357] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.389277 1170 raft_consensus.cc:383] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.389334 1170 raft_consensus.cc:738] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.389533 1170 consensus_queue.cc:260] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.390084 1170 ts_tablet_manager.cc:1428] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.083s user 0.002s sys 0.000s
I20250905 19:52:35.391187 1170 tablet_bootstrap.cc:492] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.392886 1170 tablet_bootstrap.cc:654] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.397382 2379 ts_tablet_manager.cc:1918] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.397516 2379 log.cc:1199] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/37894e39e6f2417fbfcd6d43f1ea2b37
I20250905 19:52:35.397871 2379 ts_tablet_manager.cc:1939] T 37894e39e6f2417fbfcd6d43f1ea2b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.398191 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 130) took 3814 ms. Trace:
I20250905 19:52:35.398283 2379 rpcz_store.cc:276] 0905 19:52:31.583601 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.583655 (+ 54us) service_pool.cc:225] Handling call
0905 19:52:35.309321 (+3725666us) ts_tablet_manager.cc:1145] Deleting tablet 37894e39e6f2417fbfcd6d43f1ea2b37
0905 19:52:35.309329 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.393897 (+ 84568us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.397267 (+ 3370us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.398170 (+ 903us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":74496,"tablet-delete.queue_time_us":3725491}
I20250905 19:52:35.398749 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 131) took 3807 ms. Trace:
I20250905 19:52:35.398944 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (table default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]) successfully deleted
W20250905 19:52:35.401268 849 leader_election.cc:343] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 24bcd21f19ba465aa550102e0fccb674
I20250905 19:52:35.398828 2379 rpcz_store.cc:276] 0905 19:52:31.591263 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.591311 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:35.398599 (+3807288us) ts_tablet_manager.cc:1145] Deleting tablet fd1280fefc3e43db9928567a734644f1
0905 19:52:35.398622 (+ 23us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.398729 (+ 107us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3807068}
I20250905 19:52:35.401693 2379 tablet_replica.cc:331] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.401911 2379 raft_consensus.cc:2241] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:35.401970 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd1280fefc3e43db9928567a734644f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1280fefc3e43db9928567a734644f1 already in progress: creating tablet
I20250905 19:52:35.402068 2379 raft_consensus.cc:2270] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.404102 2379 ts_tablet_manager.cc:1905] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.414589 1047 raft_consensus.cc:2466] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:35.426942 1170 tablet_bootstrap.cc:492] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.427101 1170 ts_tablet_manager.cc:1397] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.003s
I20250905 19:52:35.427789 1170 raft_consensus.cc:357] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.427989 1170 raft_consensus.cc:383] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.428041 1170 raft_consensus.cc:738] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.428241 1170 consensus_queue.cc:260] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.428761 1170 ts_tablet_manager.cc:1428] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:35.429039 1170 tablet_bootstrap.cc:492] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.430539 1170 tablet_bootstrap.cc:654] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.434159 2379 ts_tablet_manager.cc:1918] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.434273 2379 log.cc:1199] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a1b4989f17454aa58c05727b797c2009
I20250905 19:52:35.434545 2379 ts_tablet_manager.cc:1939] T a1b4989f17454aa58c05727b797c2009 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.435324 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 132) took 3824 ms. Trace:
I20250905 19:52:35.435422 2379 rpcz_store.cc:276] 0905 19:52:31.610949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.611011 (+ 62us) service_pool.cc:225] Handling call
0905 19:52:35.401642 (+3790631us) ts_tablet_manager.cc:1145] Deleting tablet a1b4989f17454aa58c05727b797c2009
0905 19:52:35.401654 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.426084 (+ 24430us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.434057 (+ 7973us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.435303 (+ 1246us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3790389}
I20250905 19:52:35.435766 2379 tablet_replica.cc:331] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.436053 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a1b4989f17454aa58c05727b797c2009 (table default.Table_ee3eaf9eb84d435187c73a394308861b [id=8225e2fcccbf436db5bfa9e05ba88f83]) successfully deleted
I20250905 19:52:35.441948 1169 tablet_bootstrap.cc:654] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.448925 1170 tablet_bootstrap.cc:492] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.449084 1170 ts_tablet_manager.cc:1397] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.020s user 0.002s sys 0.000s
I20250905 19:52:35.449812 1170 raft_consensus.cc:357] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.450003 1170 raft_consensus.cc:383] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.450064 1170 raft_consensus.cc:738] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.450277 1170 consensus_queue.cc:260] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.450733 2924 raft_consensus.cc:491] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.450788 1170 ts_tablet_manager.cc:1428] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.450870 2924 raft_consensus.cc:513] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.451062 1170 tablet_bootstrap.cc:492] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.451423 2924 leader_election.cc:290] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.452159 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:35.452654 1170 tablet_bootstrap.cc:654] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.456018 849 leader_election.cc:304] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.456110 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "421b28d047124222ae9cbc603ec22cee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.456290 2422 raft_consensus.cc:2747] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20250905 19:52:35.456622 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 55382.2us
W20250905 19:52:35.456676 984 leader_election.cc:343] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.456857 849 leader_election.cc:304] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.457098 2422 raft_consensus.cc:2747] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:35.457414 769 raft_consensus.cc:3058] T 8e0c478efe044ad0867e47f49374a66b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:35.458436 849 leader_election.cc:343] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.458554 849 leader_election.cc:304] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.416388 2413 raft_consensus.cc:491] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.421758 1171 tablet_bootstrap.cc:492] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.452010 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "421b28d047124222ae9cbc603ec22cee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.460435 1171 ts_tablet_manager.cc:1397] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.097s user 0.003s sys 0.003s
I20250905 19:52:35.460428 907 raft_consensus.cc:2466] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.460433 2413 raft_consensus.cc:513] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.461246 986 leader_election.cc:304] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.461251 1171 raft_consensus.cc:357] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.461403 1171 raft_consensus.cc:383] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.461452 1171 raft_consensus.cc:738] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.461686 2924 raft_consensus.cc:2802] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.461632 1171 consensus_queue.cc:260] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.461766 2924 raft_consensus.cc:491] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.461831 2924 raft_consensus.cc:3058] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.462132 1171 ts_tablet_manager.cc:1428] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.462421 1171 tablet_bootstrap.cc:492] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.462771 2413 leader_election.cc:290] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.463919 1171 tablet_bootstrap.cc:654] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.464080 771 raft_consensus.cc:3058] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.464942 2924 raft_consensus.cc:513] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.465992 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "421b28d047124222ae9cbc603ec22cee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.466137 907 raft_consensus.cc:3058] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.468324 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.468703 907 raft_consensus.cc:2466] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.468853 1170 tablet_bootstrap.cc:492] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.468976 1170 ts_tablet_manager.cc:1397] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20250905 19:52:35.469380 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "421b28d047124222ae9cbc603ec22cee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:35.469864 984 leader_election.cc:343] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.469973 984 leader_election.cc:304] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.470646 2931 raft_consensus.cc:2802] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.470783 2931 raft_consensus.cc:695] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
W20250905 19:52:35.470836 848 leader_election.cc:343] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.471047 2931 consensus_queue.cc:237] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.472988 1170 raft_consensus.cc:357] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.473161 1170 raft_consensus.cc:383] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.473219 1170 raft_consensus.cc:738] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.473332 2924 leader_election.cc:290] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.473394 1170 consensus_queue.cc:260] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.473889 1170 ts_tablet_manager.cc:1428] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:35.474172 1170 tablet_bootstrap.cc:492] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.474442 2924 raft_consensus.cc:491] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.474872 2924 raft_consensus.cc:513] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.477420 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2568b9f6f37b48efbe92cef4c19d5dfb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.478065 984 leader_election.cc:343] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.480697 1171 tablet_bootstrap.cc:492] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.480842 1171 ts_tablet_manager.cc:1397] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.004s
I20250905 19:52:35.481360 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2568b9f6f37b48efbe92cef4c19d5dfb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.481573 907 raft_consensus.cc:2466] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.481552 1171 raft_consensus.cc:357] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.481742 1171 raft_consensus.cc:383] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.481806 1171 raft_consensus.cc:738] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.482015 1171 consensus_queue.cc:260] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.482522 1171 ts_tablet_manager.cc:1428] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:35.482796 1171 tablet_bootstrap.cc:492] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.481114 2924 leader_election.cc:290] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.482008 986 leader_election.cc:304] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.482803 2930 raft_consensus.cc:491] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.483906 2930 raft_consensus.cc:513] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.484406 1171 tablet_bootstrap.cc:654] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.484572 2930 leader_election.cc:290] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.484699 2930 raft_consensus.cc:2802] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.484753 2930 raft_consensus.cc:491] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.484808 2930 raft_consensus.cc:3058] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.484946 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be1315824fd749049a62af3e6a113b9a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.485420 984 leader_election.cc:343] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.487558 2930 raft_consensus.cc:513] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.488822 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be1315824fd749049a62af3e6a113b9a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.489012 906 raft_consensus.cc:2466] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.489427 986 leader_election.cc:304] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.489812 2496 raft_consensus.cc:2802] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.489892 2496 raft_consensus.cc:491] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.489965 2496 raft_consensus.cc:3058] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.491096 1170 tablet_bootstrap.cc:654] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.492851 2496 raft_consensus.cc:513] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.493523 2496 leader_election.cc:290] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.494223 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be1315824fd749049a62af3e6a113b9a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:35.494261 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be1315824fd749049a62af3e6a113b9a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.494434 906 raft_consensus.cc:3058] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:35.494740 984 leader_election.cc:343] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.496867 1170 tablet_bootstrap.cc:492] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.496991 1170 ts_tablet_manager.cc:1397] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.023s user 0.002s sys 0.001s
I20250905 19:52:35.497468 906 raft_consensus.cc:2466] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.497690 1170 raft_consensus.cc:357] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.497865 1170 raft_consensus.cc:383] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.497910 1170 raft_consensus.cc:738] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.497952 986 leader_election.cc:304] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.498097 1170 consensus_queue.cc:260] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.498548 1170 ts_tablet_manager.cc:1428] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:35.498809 1170 tablet_bootstrap.cc:492] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.498828 2496 raft_consensus.cc:2802] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.498925 2496 raft_consensus.cc:695] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:35.499166 2496 consensus_queue.cc:237] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.499647 2652 ts_tablet_manager.cc:1918] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250905 19:52:35.500569 2422 raft_consensus.cc:491] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.500689 2422 raft_consensus.cc:513] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.501056 1171 tablet_bootstrap.cc:492] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.501070 774 raft_consensus.cc:3058] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.501169 1171 ts_tablet_manager.cc:1397] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.000s
I20250905 19:52:35.501241 2422 leader_election.cc:290] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.501785 1171 raft_consensus.cc:357] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.501935 1171 raft_consensus.cc:383] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.501978 1171 raft_consensus.cc:738] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.502135 1171 consensus_queue.cc:260] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.502718 1171 ts_tablet_manager.cc:1428] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 19:52:35.502892 2930 leader_election.cc:290] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.502966 1171 tablet_bootstrap.cc:492] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.503295 2930 consensus_queue.cc:1035] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:35.504451 1171 tablet_bootstrap.cc:654] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.504611 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.506685 848 leader_election.cc:343] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.506927 2413 consensus_queue.cc:1035] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:35.517700 1170 tablet_bootstrap.cc:654] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.520018 1171 tablet_bootstrap.cc:492] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.520313 1171 ts_tablet_manager.cc:1397] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250905 19:52:35.521800 1171 raft_consensus.cc:357] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.522363 2924 raft_consensus.cc:491] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.522557 2924 raft_consensus.cc:513] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.522495 1171 raft_consensus.cc:383] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.522778 1171 raft_consensus.cc:738] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.523674 2924 leader_election.cc:290] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.523869 1171 consensus_queue.cc:260] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:35.524837 848 consensus_peers.cc:489] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.527096 1171 ts_tablet_manager.cc:1428] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20250905 19:52:35.528793 1171 tablet_bootstrap.cc:492] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.535133 2492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Bootstrap replaying log segment 1/1 (232.2K/265.8K this segment, stats: ops{read=518 overwritten=0 applied=511 ignored=0} inserts{seen=463 ignored=0} mutations{seen=590 ignored=0} orphaned_commits=0)
I20250905 19:52:35.536361 1170 tablet_bootstrap.cc:492] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.536514 1170 ts_tablet_manager.cc:1397] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.038s user 0.007s sys 0.000s
I20250905 19:52:35.537472 1170 raft_consensus.cc:357] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.537674 1170 raft_consensus.cc:383] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.537727 1170 raft_consensus.cc:738] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.537932 1170 consensus_queue.cc:260] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.538020 1171 tablet_bootstrap.cc:654] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.538331 2376 ts_tablet_manager.cc:1918] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.538691 1170 ts_tablet_manager.cc:1428] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.538679 2376 log.cc:1199] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:35.538964 1170 tablet_bootstrap.cc:492] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.539251 2379 raft_consensus.cc:2241] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.539723 2379 pending_rounds.cc:70] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
I20250905 19:52:35.539772 2379 pending_rounds.cc:77] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089609263861760 op_type: NO_OP noop_request { }
I20250905 19:52:35.539853 2379 raft_consensus.cc:2887] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:35.539929 2379 raft_consensus.cc:2270] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.540730 1170 tablet_bootstrap.cc:654] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.544185 2789 raft_consensus.cc:491] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.544400 2789 raft_consensus.cc:513] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.545011 2789 leader_election.cc:290] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.545640 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd1280fefc3e43db9928567a734644f1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.546150 848 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.546402 1849 raft_consensus.cc:491] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.546572 1849 raft_consensus.cc:513] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.547360 1849 leader_election.cc:290] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.550226 1169 tablet_bootstrap.cc:492] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:35.550400 1169 ts_tablet_manager.cc:1397] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.215s user 0.003s sys 0.004s
I20250905 19:52:35.551215 1169 raft_consensus.cc:357] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.551888 1169 raft_consensus.cc:383] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.551977 1169 raft_consensus.cc:738] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.552353 2652 log.cc:1199] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8edf4b54974c427a855eee5792d75292
I20250905 19:52:35.552904 2652 ts_tablet_manager.cc:1939] T 8edf4b54974c427a855eee5792d75292 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:35.554433 1170 tablet_bootstrap.cc:492] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.554603 1170 ts_tablet_manager.cc:1397] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.000s
I20250905 19:52:35.555387 1170 raft_consensus.cc:357] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.555572 1170 raft_consensus.cc:383] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.555641 1170 raft_consensus.cc:738] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.555673 2379 ts_tablet_manager.cc:1905] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.555856 1170 consensus_queue.cc:260] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.561957 1170 ts_tablet_manager.cc:1428] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20250905 19:52:35.562269 1170 tablet_bootstrap.cc:492] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.562719 2379 ts_tablet_manager.cc:1918] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.562837 2379 log.cc:1199] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:35.563252 2379 ts_tablet_manager.cc:1939] T 52c99a76802b481c98f77dbd90de7cbb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.563597 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 133) took 3950 ms. Trace:
I20250905 19:52:35.563696 2379 rpcz_store.cc:276] 0905 19:52:31.612899 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.612973 (+ 74us) service_pool.cc:225] Handling call
0905 19:52:35.435716 (+3822743us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:35.435726 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.539556 (+103830us) spinlock_profiling.cc:243] Waited 103 ms on lock 0x616000292054. stack: 00007fb0d72d6dc6 00007fb0d72d6b80 00007fb0d62be3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fb0e2d99600 00007fb0e40acb13 00007fb0e6b8f2b7 00007fb0e6bc1344 00007fb0e6ba2da0 00007fb0e6ba2b2d 0000000000a774dc 00007fb0d731d14f 00007fb0d732041b 00007fb0d7320298
0905 19:52:35.559315 (+ 19759us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.562604 (+ 3289us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.563576 (+ 972us) inbound_call.cc:173] Queueing success response
Metrics: {"spinlock_wait_cycles":237577472,"tablet-delete.queue_time_us":3822536}
I20250905 19:52:35.564476 2379 tablet_replica.cc:331] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.564523 1170 tablet_bootstrap.cc:654] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.564759 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 52c99a76802b481c98f77dbd90de7cbb (table default.Table_29345948e72a432f9d41b63ca0c45110 [id=7ae4e86b2cb6498f90c5a0f9c662a82f]) successfully deleted
I20250905 19:52:35.565260 2379 raft_consensus.cc:2241] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.565522 2379 raft_consensus.cc:2270] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.566262 2379 ts_tablet_manager.cc:1905] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.568871 1171 tablet_bootstrap.cc:492] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.569151 1171 ts_tablet_manager.cc:1397] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.041s user 0.008s sys 0.001s
I20250905 19:52:35.570644 1171 raft_consensus.cc:357] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.570971 1171 raft_consensus.cc:383] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.571100 1171 raft_consensus.cc:738] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.571463 1171 consensus_queue.cc:260] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.572913 1171 ts_tablet_manager.cc:1428] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:35.574265 1171 tablet_bootstrap.cc:492] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.582145 1171 tablet_bootstrap.cc:654] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.587615 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:35.588205 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 85628.3us
I20250905 19:52:35.588932 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2568b9f6f37b48efbe92cef4c19d5dfb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.589138 907 raft_consensus.cc:3058] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.592034 907 raft_consensus.cc:2466] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.597257 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 75) took 4021 ms. Trace:
I20250905 19:52:35.597431 2652 rpcz_store.cc:276] 0905 19:52:31.575571 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.575735 (+ 164us) service_pool.cc:225] Handling call
0905 19:52:35.206109 (+3630374us) ts_tablet_manager.cc:1145] Deleting tablet 8edf4b54974c427a855eee5792d75292
0905 19:52:35.206119 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.392547 (+186428us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.499377 (+106830us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.597236 (+ 97859us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3629708}
I20250905 19:52:35.598038 2652 tablet_replica.cc:331] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:35.598459 2652 raft_consensus.cc:2241] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.598912 2652 raft_consensus.cc:2270] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.598966 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8edf4b54974c427a855eee5792d75292 (table default.Table_93a9ab7d352a4a3da35219aa6f20f432 [id=b5acc86aee9545d99f8d8332c3c76a8e]) successfully deleted
I20250905 19:52:35.600236 2931 raft_consensus.cc:491] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.600381 2931 raft_consensus.cc:513] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.600955 2931 leader_election.cc:290] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.601241 1171 tablet_bootstrap.cc:492] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.601523 1171 ts_tablet_manager.cc:1397] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.028s user 0.009s sys 0.000s
I20250905 19:52:35.602843 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7155cf5abba4e2cbdce583406f94fad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.603013 907 raft_consensus.cc:2466] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.603024 1171 raft_consensus.cc:357] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.603497 1171 raft_consensus.cc:383] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.603694 1171 raft_consensus.cc:738] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.604226 1171 consensus_queue.cc:260] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.605149 2376 ts_tablet_manager.cc:1939] T fd793bce71e344f18bb177a53ab1bfc1 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:35.605305 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.605500 906 raft_consensus.cc:2466] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.605715 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 83) took 6156 ms. Trace:
I20250905 19:52:35.605908 986 leader_election.cc:304] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:35.606316 986 leader_election.cc:304] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:35.606034 2376 rpcz_store.cc:276] 0905 19:52:29.448774 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.448899 (+ 125us) service_pool.cc:225] Handling call
0905 19:52:35.342169 (+5893270us) ts_tablet_manager.cc:1145] Deleting tablet fd793bce71e344f18bb177a53ab1bfc1
0905 19:52:35.342182 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.418650 (+ 76468us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.538077 (+119427us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.605636 (+ 67559us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":477,"tablet-delete.queue_time_us":5892167}
I20250905 19:52:35.606930 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet fd793bce71e344f18bb177a53ab1bfc1 (table default.Table_a414c05c2c0c40b8a55a7776f36ecdb7 [id=1e2786d48c8d49cfa9cbca17c164b7eb]) successfully deleted
I20250905 19:52:35.607014 2376 tablet_replica.cc:331] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:35.607748 1171 ts_tablet_manager.cc:1428] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20250905 19:52:35.608108 2376 raft_consensus.cc:2241] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.609184 1171 tablet_bootstrap.cc:492] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.580435 1170 tablet_bootstrap.cc:492] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.585867 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7155cf5abba4e2cbdce583406f94fad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.609629 1170 ts_tablet_manager.cc:1397] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.047s user 0.004s sys 0.000s
W20250905 19:52:35.610235 848 consensus_peers.cc:489] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.610270 1170 raft_consensus.cc:357] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.610440 1170 raft_consensus.cc:383] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.610503 1170 raft_consensus.cc:738] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.610713 1170 consensus_queue.cc:260] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.611189 1170 ts_tablet_manager.cc:1428] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:35.611454 1170 tablet_bootstrap.cc:492] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.613291 1170 tablet_bootstrap.cc:654] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.614638 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.603356 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:35.609031 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2568b9f6f37b48efbe92cef4c19d5dfb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:35.619477 848 consensus_peers.cc:489] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.622125 1171 tablet_bootstrap.cc:654] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.624353 986 leader_election.cc:304] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:35.626434 2376 raft_consensus.cc:2270] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.628163 2496 raft_consensus.cc:2802] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.628340 2496 raft_consensus.cc:491] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.628479 2496 raft_consensus.cc:3058] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:35.629141 848 consensus_peers.cc:489] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.629452 1170 tablet_bootstrap.cc:492] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.629604 1170 ts_tablet_manager.cc:1397] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.000s
I20250905 19:52:35.630319 1170 raft_consensus.cc:357] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.630496 1170 raft_consensus.cc:383] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.630559 1170 raft_consensus.cc:738] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.630770 1170 consensus_queue.cc:260] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.631553 1170 ts_tablet_manager.cc:1428] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.631844 1170 tablet_bootstrap.cc:492] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.634076 1170 tablet_bootstrap.cc:654] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.608575 2998 raft_consensus.cc:2802] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.634637 2998 raft_consensus.cc:695] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:35.635054 2998 consensus_queue.cc:237] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.624706 2931 raft_consensus.cc:2802] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.636401 2931 raft_consensus.cc:491] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.636543 2931 raft_consensus.cc:3058] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.552290 1169 consensus_queue.cc:260] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.637602 1169 ts_tablet_manager.cc:1428] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.087s user 0.002s sys 0.000s
I20250905 19:52:35.638299 1169 tablet_bootstrap.cc:492] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:35.625527 984 leader_election.cc:343] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:35.638959 984 leader_election.cc:343] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:35.639379 984 leader_election.cc:343] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:35.639693 984 consensus_peers.cc:489] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.639856 1099 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:35.640780 2376 ts_tablet_manager.cc:1905] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.641057 2996 raft_consensus.cc:491] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:35.641216 2996 raft_consensus.cc:513] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.641626 1171 tablet_bootstrap.cc:492] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.641855 1171 ts_tablet_manager.cc:1397] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.033s user 0.005s sys 0.001s
I20250905 19:52:35.641928 2996 leader_election.cc:290] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.643265 1171 raft_consensus.cc:357] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.643610 1171 raft_consensus.cc:383] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.643815 1171 raft_consensus.cc:738] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.644404 1171 consensus_queue.cc:260] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.646708 1171 ts_tablet_manager.cc:1428] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20250905 19:52:35.647946 1171 tablet_bootstrap.cc:492] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:35.648286 984 leader_election.cc:343] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 9e9994863c374354838aa0f817d0057e
I20250905 19:52:35.648355 1170 tablet_bootstrap.cc:492] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.648476 1170 ts_tablet_manager.cc:1397] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.002s
I20250905 19:52:35.600242 2652 ts_tablet_manager.cc:1905] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.649216 1170 raft_consensus.cc:357] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.649385 1170 raft_consensus.cc:383] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.649443 1170 raft_consensus.cc:738] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.649669 1170 consensus_queue.cc:260] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.650120 1170 ts_tablet_manager.cc:1428] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.654222 1171 tablet_bootstrap.cc:654] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.662433 2496 raft_consensus.cc:513] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.666864 2496 leader_election.cc:290] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.667531 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:35.667948 1169 tablet_bootstrap.cc:654] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:35.669165 984 leader_election.cc:343] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.671114 1171 tablet_bootstrap.cc:492] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.671394 1171 ts_tablet_manager.cc:1397] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.000s sys 0.006s
I20250905 19:52:35.665427 2931 raft_consensus.cc:513] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:35.665500 986 leader_election.cc:343] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 9e9994863c374354838aa0f817d0057e
I20250905 19:52:35.672741 986 leader_election.cc:304] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.674763 2492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Bootstrap replayed 1/1 log segments. Stats: ops{read=593 overwritten=0 applied=590 ignored=0} inserts{seen=536 ignored=0} mutations{seen=673 ignored=0} orphaned_commits=0. Pending: 3 replicates
I20250905 19:52:35.674980 2924 raft_consensus.cc:2747] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:35.675953 2492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Bootstrap complete.
I20250905 19:52:35.675886 1171 raft_consensus.cc:357] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.676244 1171 raft_consensus.cc:383] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.676391 1171 raft_consensus.cc:738] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.676792 1171 consensus_queue.cc:260] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.676978 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.677237 907 raft_consensus.cc:3058] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.677934 2931 leader_election.cc:290] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.679101 1171 ts_tablet_manager.cc:1428] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.003s sys 0.000s
I20250905 19:52:35.680408 907 raft_consensus.cc:2466] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.680748 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7155cf5abba4e2cbdce583406f94fad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:35.681239 984 leader_election.cc:343] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.681809 774 raft_consensus.cc:3058] T c4b31443b0f840749e12435e197a204a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.683059 1849 raft_consensus.cc:491] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.683173 1849 raft_consensus.cc:513] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.683684 1849 leader_election.cc:290] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.684500 1170 tablet_bootstrap.cc:492] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.686309 1170 tablet_bootstrap.cc:654] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.680406 1171 tablet_bootstrap.cc:492] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.687666 986 leader_election.cc:304] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.688978 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:35.689177 1047 raft_consensus.cc:2391] T a311f9b943d143a9a52d7771ce7186d7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:35.689268 1171 tablet_bootstrap.cc:654] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.689519 2496 raft_consensus.cc:2802] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.689795 2496 raft_consensus.cc:695] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:35.690081 2496 consensus_queue.cc:237] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.690169 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d7155cf5abba4e2cbdce583406f94fad" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.690515 907 raft_consensus.cc:3058] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.683887 2492 raft_consensus.cc:357] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 FOLLOWER]: Replica starting. Triggering 3 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:35.705610 1170 tablet_bootstrap.cc:492] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.705760 1170 ts_tablet_manager.cc:1397] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.006s
I20250905 19:52:35.706486 1170 raft_consensus.cc:357] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.706667 1170 raft_consensus.cc:383] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.706718 1170 raft_consensus.cc:738] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.706900 1170 consensus_queue.cc:260] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.707340 1170 ts_tablet_manager.cc:1428] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.707939 1170 tablet_bootstrap.cc:492] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.708123 1169 tablet_bootstrap.cc:492] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:35.708343 1169 ts_tablet_manager.cc:1397] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.001s
I20250905 19:52:35.709370 1169 raft_consensus.cc:357] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.709605 1169 raft_consensus.cc:383] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.709705 1169 raft_consensus.cc:738] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.709995 1169 consensus_queue.cc:260] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.710561 1169 ts_tablet_manager.cc:1428] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:35.710875 1169 tablet_bootstrap.cc:492] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:35.715152 1170 tablet_bootstrap.cc:654] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.720175 2379 ts_tablet_manager.cc:1918] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.720398 2379 log.cc:1199] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/cbfd03ad0bd34b6ca86e7b07a62b2a8e
I20250905 19:52:35.721097 1170 tablet_bootstrap.cc:492] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.721226 1170 ts_tablet_manager.cc:1397] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.005s
I20250905 19:52:35.722004 1170 raft_consensus.cc:357] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.722173 1170 raft_consensus.cc:383] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.722229 1170 raft_consensus.cc:738] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.722510 1170 consensus_queue.cc:260] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.723582 2379 ts_tablet_manager.cc:1939] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.723002 907 raft_consensus.cc:2466] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.723959 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 134) took 4110 ms. Trace:
I20250905 19:52:35.725139 986 leader_election.cc:304] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.725805 2931 raft_consensus.cc:2802] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.726202 2931 raft_consensus.cc:695] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:35.727455 2931 consensus_queue.cc:237] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.730798 2492 raft_consensus.cc:738] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: c401c9f3020b4ae5931c566c6bcb1711, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.731778 2492 consensus_queue.cc:260] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 590, Last appended: 2.593, Last appended by leader: 593, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:35.733093 2738 raft_consensus.cc:491] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.733227 2738 raft_consensus.cc:513] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.733801 2738 leader_election.cc:290] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.726992 1849 raft_consensus.cc:491] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.729835 1170 ts_tablet_manager.cc:1428] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20250905 19:52:35.734753 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca646f7c162e41469f283657e72823b4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.737792 1849 raft_consensus.cc:513] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.738090 1170 tablet_bootstrap.cc:492] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.737709 2379 rpcz_store.cc:276] 0905 19:52:31.613674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.613721 (+ 47us) service_pool.cc:225] Handling call
0905 19:52:35.564432 (+3950711us) ts_tablet_manager.cc:1145] Deleting tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
0905 19:52:35.564441 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.613260 (+ 48819us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.720046 (+106786us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.723943 (+ 3897us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3950453}
I20250905 19:52:35.738344 1849 leader_election.cc:290] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.738734 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e (table default.Table_c7c4cd0bb1b14ead83f2942518f4c30f [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
W20250905 19:52:35.738960 848 leader_election.cc:343] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.739223 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:35.740172 1170 tablet_bootstrap.cc:654] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.741375 1169 tablet_bootstrap.cc:654] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.742633 2652 ts_tablet_manager.cc:1918] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.742832 2652 log.cc:1199] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a5af8232d31240ff855b79dcea29396b
I20250905 19:52:35.743499 2376 ts_tablet_manager.cc:1918] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.744462 2376 log.cc:1199] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:35.744694 771 raft_consensus.cc:3058] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.746295 2376 ts_tablet_manager.cc:1939] T 5bd9380e8ffc432da4a684934c2df68f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:35.748134 2738 consensus_queue.cc:1035] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:35.748773 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 85) took 6297 ms. Trace:
I20250905 19:52:35.749284 2376 rpcz_store.cc:276] 0905 19:52:29.449998 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.451356 (+ 1358us) service_pool.cc:225] Handling call
0905 19:52:35.606815 (+6155459us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:35.606871 (+ 56us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.703261 (+ 96390us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.743080 (+ 39819us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.747761 (+ 4681us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6155139}
I20250905 19:52:35.751904 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 5bd9380e8ffc432da4a684934c2df68f (table default.Table_a21b18f7481c4713823fadf4b164b7bf [id=16bc518342ba416b9ef21427ac1429f8]) successfully deleted
I20250905 19:52:35.755023 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 135) took 4140 ms. Trace:
I20250905 19:52:35.755136 2379 rpcz_store.cc:276] 0905 19:52:31.614535 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.614578 (+ 43us) service_pool.cc:225] Handling call
0905 19:52:35.738279 (+4123701us) ts_tablet_manager.cc:1145] Deleting tablet 36f966bf512d4cbe9ae406386db31533
0905 19:52:35.738288 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.754998 (+ 16710us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4123506}
I20250905 19:52:35.755548 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 136) took 4138 ms. Trace:
I20250905 19:52:35.755633 2379 rpcz_store.cc:276] 0905 19:52:31.617387 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.617433 (+ 46us) service_pool.cc:225] Handling call
0905 19:52:35.755423 (+4137990us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:35.755434 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.755528 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4137810}
W20250905 19:52:35.755729 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 36f966bf512d4cbe9ae406386db31533 because the tablet was not found. No further retry: Not found: Tablet not found: 36f966bf512d4cbe9ae406386db31533
I20250905 19:52:35.755900 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 137) took 4137 ms. Trace:
W20250905 19:52:35.756428 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 14d5eb8bec074d2c8766fe7b159031e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14d5eb8bec074d2c8766fe7b159031e5 already in progress: creating tablet
I20250905 19:52:35.754819 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 86) took 6302 ms. Trace:
I20250905 19:52:35.757670 2376 rpcz_store.cc:276] 0905 19:52:29.452223 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.452321 (+ 98us) service_pool.cc:225] Handling call
0905 19:52:35.753228 (+6300907us) ts_tablet_manager.cc:1145] Deleting tablet 813f208881ac4782874138987a710d93
0905 19:52:35.753444 (+ 216us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.754567 (+ 1123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6298765}
W20250905 19:52:35.761031 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 813f208881ac4782874138987a710d93 because the tablet was not found. No further retry: Not found: Tablet not found: 813f208881ac4782874138987a710d93
I20250905 19:52:35.762404 2652 ts_tablet_manager.cc:1939] T a5af8232d31240ff855b79dcea29396b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:35.763581 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 77) took 4164 ms. Trace:
I20250905 19:52:35.763319 2376 tablet_replica.cc:331] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:35.755959 2379 rpcz_store.cc:276] 0905 19:52:31.618201 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.618598 (+ 397us) service_pool.cc:225] Handling call
0905 19:52:35.755834 (+4137236us) ts_tablet_manager.cc:1145] Deleting tablet baefaa1f042348c1874e306f1e5a8781
0905 19:52:35.755841 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.755887 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4137044}
I20250905 19:52:35.764557 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:35.770516 2376 raft_consensus.cc:2241] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.772518 2376 raft_consensus.cc:2270] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.778481 2376 ts_tablet_manager.cc:1905] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.782214 771 raft_consensus.cc:3058] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.787232 2379 tablet_replica.cc:331] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:35.787441 2379 raft_consensus.cc:2241] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.787596 2379 raft_consensus.cc:2270] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.788183 2379 ts_tablet_manager.cc:1905] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:35.790822 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet baefaa1f042348c1874e306f1e5a8781 because the tablet was not found. No further retry: Not found: Tablet not found: baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:35.792630 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:35.793126 1171 tablet_bootstrap.cc:492] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.793457 1171 ts_tablet_manager.cc:1397] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.114s user 0.004s sys 0.007s
I20250905 19:52:35.789732 2652 rpcz_store.cc:276] 0905 19:52:31.598891 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.599208 (+ 317us) service_pool.cc:225] Handling call
0905 19:52:35.597824 (+3998616us) ts_tablet_manager.cc:1145] Deleting tablet a5af8232d31240ff855b79dcea29396b
0905 19:52:35.597833 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.687476 (+ 89643us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.742512 (+ 55036us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.763521 (+ 21009us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3981155}
I20250905 19:52:35.795269 1171 raft_consensus.cc:357] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.795747 1171 raft_consensus.cc:383] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.795890 1171 raft_consensus.cc:738] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.796419 1171 consensus_queue.cc:260] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.797829 1171 ts_tablet_manager.cc:1428] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.002s sys 0.001s
I20250905 19:52:35.798525 1171 tablet_bootstrap.cc:492] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:35.799404 775 raft_consensus.cc:3058] T 0d93a328d4934e5294ec66f739eb240b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.794682 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a5af8232d31240ff855b79dcea29396b (table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]) successfully deleted
I20250905 19:52:35.812889 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:35.813899 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:35.815833 1169 tablet_bootstrap.cc:492] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:35.816097 1169 ts_tablet_manager.cc:1397] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.105s user 0.004s sys 0.000s
I20250905 19:52:35.816854 1170 tablet_bootstrap.cc:492] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.817001 1170 ts_tablet_manager.cc:1397] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.079s user 0.010s sys 0.022s
I20250905 19:52:35.817835 1170 raft_consensus.cc:357] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.818019 1170 raft_consensus.cc:383] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.818079 1170 raft_consensus.cc:738] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.818295 1170 consensus_queue.cc:260] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.818776 1170 ts_tablet_manager.cc:1428] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.819054 1170 tablet_bootstrap.cc:492] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:35.828733 1171 tablet_bootstrap.cc:654] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.837067 1170 tablet_bootstrap.cc:654] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.842891 1169 raft_consensus.cc:357] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.843309 1169 raft_consensus.cc:383] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.843482 1169 raft_consensus.cc:738] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.843793 1169 consensus_queue.cc:260] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.844532 1169 ts_tablet_manager.cc:1428] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:35.853034 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:35.854024 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:35.854898 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:35.857960 2492 sys_catalog.cc:564] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: configured and running, proceeding with master startup.
I20250905 19:52:35.862598 2422 raft_consensus.cc:2747] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:35.862674 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 404287us
I20250905 19:52:35.867611 2376 ts_tablet_manager.cc:1918] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:35.867744 2376 log.cc:1199] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:35.868072 2376 ts_tablet_manager.cc:1939] T 52c99a76802b481c98f77dbd90de7cbb P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:35.871416 2379 ts_tablet_manager.cc:1918] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250905 19:52:35.871562 2379 log.cc:1199] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:35.872020 2379 ts_tablet_manager.cc:1939] T 98e5ca7bd3e84916a11e01afbf48171d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:35.873000 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 138) took 4253 ms. Trace:
I20250905 19:52:35.873118 2379 rpcz_store.cc:276] 0905 19:52:31.619287 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.619332 (+ 45us) service_pool.cc:225] Handling call
0905 19:52:35.787176 (+4167844us) ts_tablet_manager.cc:1145] Deleting tablet 98e5ca7bd3e84916a11e01afbf48171d
0905 19:52:35.787188 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.840828 (+ 53640us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.871265 (+ 30437us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.872974 (+ 1709us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4167667}
I20250905 19:52:35.873719 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 98e5ca7bd3e84916a11e01afbf48171d (table default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff [id=8f425647eea7481181ba823eb955d1e1]) successfully deleted
I20250905 19:52:35.874151 2440 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 651 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:35.868742 2448 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 651 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
W20250905 19:52:35.877990 2440 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43956 (request call id 3) took 1941 ms (client timeout 564 ms). Trace:
I20250905 19:52:35.880401 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 82) took 4257 ms. Trace:
I20250905 19:52:35.880527 2652 rpcz_store.cc:276] 0905 19:52:31.622410 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.622879 (+ 469us) service_pool.cc:225] Handling call
0905 19:52:35.796767 (+4173888us) ts_tablet_manager.cc:1145] Deleting tablet 03a12cf719f34096bb82f5846bbe3b2d
0905 19:52:35.880222 (+ 83455us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.880332 (+ 110us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4172318}
I20250905 19:52:35.880904 2652 tablet_replica.cc:331] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:35.881103 2652 raft_consensus.cc:2241] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:35.881316 2652 raft_consensus.cc:2270] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.881969 2652 ts_tablet_manager.cc:1905] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.884358 2838 consensus_queue.cc:1035] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:35.884595 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 03a12cf719f34096bb82f5846bbe3b2d because the tablet was not found. No further retry: Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
I20250905 19:52:35.885224 849 leader_election.cc:304] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.876004 2451 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 636 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:35.888370 2930 raft_consensus.cc:491] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.888559 2930 raft_consensus.cc:513] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.889135 2930 leader_election.cc:290] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.889745 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d495b6ba92e84b588b4780c2896195c0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:35.889904 906 raft_consensus.cc:2466] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:35.890535 1171 tablet_bootstrap.cc:492] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.890587 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d495b6ba92e84b588b4780c2896195c0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:35.890690 1171 ts_tablet_manager.cc:1397] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.002s
I20250905 19:52:35.891031 986 leader_election.cc:304] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:35.891350 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 93) took 6437 ms. Trace:
I20250905 19:52:35.891434 2376 rpcz_store.cc:276] 0905 19:52:29.454070 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.454118 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:35.761752 (+6307634us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:35.761764 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.837815 (+ 76051us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.867164 (+ 29349us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.891331 (+ 24167us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22454,"tablet-delete.queue_time_us":6307194}
W20250905 19:52:35.891608 984 leader_election.cc:343] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.891805 2376 tablet_replica.cc:331] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:35.891723 1171 raft_consensus.cc:357] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.891916 1171 raft_consensus.cc:383] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.891992 1171 raft_consensus.cc:738] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.892032 2376 raft_consensus.cc:2241] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.892166 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 52c99a76802b481c98f77dbd90de7cbb (table default.Table_29345948e72a432f9d41b63ca0c45110 [id=7ae4e86b2cb6498f90c5a0f9c662a82f]) successfully deleted
I20250905 19:52:35.892318 2376 pending_rounds.cc:70] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:35.892233 1171 consensus_queue.cc:260] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.892360 2376 pending_rounds.cc:77] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089593107148800 op_type: NO_OP noop_request { }
W20250905 19:52:35.885615 2440 rpcz_store.cc:269] 0905 19:52:33.936111 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:33.936172 (+ 61us) service_pool.cc:225] Handling call
0905 19:52:35.877934 (+1941762us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:35.893299 2930 raft_consensus.cc:2802] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:35.893396 2930 raft_consensus.cc:491] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:35.893503 2930 raft_consensus.cc:3058] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.893486 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
W20250905 19:52:35.878556 2448 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43956 (request call id 2) took 2533 ms (client timeout 564 ms). Trace:
W20250905 19:52:35.894019 2448 rpcz_store.cc:269] 0905 19:52:33.345188 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:33.345245 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:35.878487 (+2533242us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:35.894243 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]). Expected version 2 got 0
I20250905 19:52:35.894487 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:35.876113 2450 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 636 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
W20250905 19:52:35.893374 2451 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43956 (request call id 0) took 5466 ms (client timeout 507 ms). Trace:
I20250905 19:52:35.894817 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
W20250905 19:52:35.894824 2451 rpcz_store.cc:269] 0905 19:52:30.426570 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:30.426643 (+ 73us) service_pool.cc:225] Handling call
0905 19:52:35.893327 (+5466684us) inbound_call.cc:173] Queueing success response
Metrics: {}
W20250905 19:52:35.895112 2450 rpcz_store.cc:267] Call kudu.consensus.ConsensusService.RequestConsensusVote from 127.0.0.1:43956 (request call id 1) took 5092 ms (client timeout 507 ms). Trace:
I20250905 19:52:35.895619 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
W20250905 19:52:35.895627 2450 rpcz_store.cc:269] 0905 19:52:30.803108 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:30.803179 (+ 71us) service_pool.cc:225] Handling call
0905 19:52:35.895077 (+5091898us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:35.895839 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:35.896142 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:35.896508 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:35.892465 2376 raft_consensus.cc:2887] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:35.896735 2376 raft_consensus.cc:2270] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:35.897312 1169 tablet_bootstrap.cc:492] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:35.897341 2376 ts_tablet_manager.cc:1905] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.897338 2452 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 2 FOLLOWER]: Advancing to term 3
I20250905 19:52:35.899434 1169 tablet_bootstrap.cc:654] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.900830 2930 raft_consensus.cc:513] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.901803 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d495b6ba92e84b588b4780c2896195c0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:35.902312 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d495b6ba92e84b588b4780c2896195c0" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:35.902460 906 raft_consensus.cc:3058] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:35.903810 1171 ts_tablet_manager.cc:1428] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20250905 19:52:35.904219 1171 tablet_bootstrap.cc:492] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:35.905100 984 leader_election.cc:343] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.905448 2930 leader_election.cc:290] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:35.906368 1171 tablet_bootstrap.cc:654] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:35.907370 906 raft_consensus.cc:2466] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:35.907835 986 leader_election.cc:304] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:35.908649 2930 raft_consensus.cc:2802] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.908921 2930 raft_consensus.cc:695] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:35.909214 2930 consensus_queue.cc:237] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.912101 1170 tablet_bootstrap.cc:492] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:35.912252 1170 ts_tablet_manager.cc:1397] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.093s user 0.005s sys 0.029s
I20250905 19:52:35.913183 1170 raft_consensus.cc:357] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.913371 1170 raft_consensus.cc:383] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.913424 1170 raft_consensus.cc:738] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.913614 1170 consensus_queue.cc:260] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.914093 1170 ts_tablet_manager.cc:1428] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:35.914703 2652 ts_tablet_manager.cc:1918] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:35.915438 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:35.916388 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:35.916998 2652 log.cc:1199] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:35.922600 1849 raft_consensus.cc:491] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4eb3af79b01d4835bf6bf7b26bf016f5)
I20250905 19:52:35.922796 1849 raft_consensus.cc:513] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.923637 1849 leader_election.cc:290] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.925067 1170 tablet_bootstrap.cc:492] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:35.925166 711 leader_election.cc:343] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:35.925829 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 139) took 4305 ms. Trace:
I20250905 19:52:35.925995 2652 ts_tablet_manager.cc:1939] T ac5c88cf8d15433a940e937f0a225582 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:35.926000 2379 rpcz_store.cc:276] 0905 19:52:31.619949 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.620010 (+ 61us) service_pool.cc:225] Handling call
0905 19:52:35.914339 (+4294329us) ts_tablet_manager.cc:1145] Deleting tablet 0edac849e8834894be2d4ff323edd13d
0905 19:52:35.925729 (+ 11390us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.925810 (+ 81us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4294167}
I20250905 19:52:35.926764 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 140) took 4305 ms. Trace:
I20250905 19:52:35.926892 2379 rpcz_store.cc:276] 0905 19:52:31.620846 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.620892 (+ 46us) service_pool.cc:225] Handling call
0905 19:52:35.926668 (+4305776us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6109bdf8f4caf8c223a42401b94ab
0905 19:52:35.926677 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.926731 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4305586}
I20250905 19:52:35.927152 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 80) took 4304 ms. Trace:
I20250905 19:52:35.927325 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 141) took 4293 ms. Trace:
I20250905 19:52:35.927428 2379 rpcz_store.cc:276] 0905 19:52:31.634205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.634260 (+ 55us) service_pool.cc:225] Handling call
0905 19:52:35.927217 (+4292957us) ts_tablet_manager.cc:1145] Deleting tablet 42f3c1efb94e446fb0128b6b412821e2
0905 19:52:35.927227 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.927280 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4292738}
I20250905 19:52:35.927798 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 142) took 4292 ms. Trace:
I20250905 19:52:35.927901 2379 rpcz_store.cc:276] 0905 19:52:31.635191 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.635241 (+ 50us) service_pool.cc:225] Handling call
0905 19:52:35.927706 (+4292465us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:35.927714 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.927759 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4291987}
I20250905 19:52:35.928339 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 144) took 4274 ms. Trace:
I20250905 19:52:35.928443 2379 rpcz_store.cc:276] 0905 19:52:31.653770 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.653852 (+ 82us) service_pool.cc:225] Handling call
0905 19:52:35.928208 (+4274356us) ts_tablet_manager.cc:1145] Deleting tablet 2f581403ec924fb0948400df41ed8aab
0905 19:52:35.928219 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.928305 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274148}
I20250905 19:52:35.928887 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 143) took 4275 ms. Trace:
I20250905 19:52:35.928989 2379 rpcz_store.cc:276] 0905 19:52:31.653613 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.653840 (+ 227us) service_pool.cc:225] Handling call
0905 19:52:35.928788 (+4274948us) ts_tablet_manager.cc:1145] Deleting tablet 757a7d3fba8d43dda36474539b7296ba
0905 19:52:35.928799 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.928852 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274683}
I20250905 19:52:35.929391 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 145) took 4274 ms. Trace:
I20250905 19:52:35.929494 2379 rpcz_store.cc:276] 0905 19:52:31.654554 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.654603 (+ 49us) service_pool.cc:225] Handling call
0905 19:52:35.929291 (+4274688us) ts_tablet_manager.cc:1145] Deleting tablet 0267ac3e3f1a45978807737eca5d91ef
0905 19:52:35.929309 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.929360 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274521}
I20250905 19:52:35.929981 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 146) took 4274 ms. Trace:
I20250905 19:52:35.930086 2379 rpcz_store.cc:276] 0905 19:52:31.655417 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.655522 (+ 105us) service_pool.cc:225] Handling call
0905 19:52:35.929889 (+4274367us) ts_tablet_manager.cc:1145] Deleting tablet 00d8144bb4f14358af539b7aae326918
0905 19:52:35.929897 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.929943 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4274204}
I20250905 19:52:35.930466 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 147) took 4274 ms. Trace:
I20250905 19:52:35.930569 2379 rpcz_store.cc:276] 0905 19:52:31.656350 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.656391 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:35.930370 (+4273979us) ts_tablet_manager.cc:1145] Deleting tablet 0d4204f61929498188c18529148a8d13
0905 19:52:35.930378 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.930428 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273826}
I20250905 19:52:35.930943 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 148) took 4273 ms. Trace:
I20250905 19:52:35.931103 2379 rpcz_store.cc:276] 0905 19:52:31.657459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.657500 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:35.930853 (+4273353us) ts_tablet_manager.cc:1145] Deleting tablet 60a41dbd8782460cb0d3c5f8bd59354f
0905 19:52:35.930861 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.930906 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273201}
I20250905 19:52:35.931495 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 149) took 4273 ms. Trace:
I20250905 19:52:35.931597 2379 rpcz_store.cc:276] 0905 19:52:31.658169 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.658210 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:35.931405 (+4273195us) ts_tablet_manager.cc:1145] Deleting tablet 4d2ed1bc09c94526bc9662f2d4d21f8f
0905 19:52:35.931413 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.931459 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4273019}
W20250905 19:52:35.926504 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0edac849e8834894be2d4ff323edd13d because the tablet was not found. No further retry: Not found: Tablet not found: 0edac849e8834894be2d4ff323edd13d
W20250905 19:52:35.932155 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8ed6109bdf8f4caf8c223a42401b94ab because the tablet was not found. No further retry: Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab
W20250905 19:52:35.932400 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 42f3c1efb94e446fb0128b6b412821e2 because the tablet was not found. No further retry: Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2
W20250905 19:52:35.932593 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a8b6823580cb4040b49ed0a49b748965 because the tablet was not found. No further retry: Not found: Tablet not found: a8b6823580cb4040b49ed0a49b748965
W20250905 19:52:35.932775 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2f581403ec924fb0948400df41ed8aab because the tablet was not found. No further retry: Not found: Tablet not found: 2f581403ec924fb0948400df41ed8aab
W20250905 19:52:35.932943 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 757a7d3fba8d43dda36474539b7296ba because the tablet was not found. No further retry: Not found: Tablet not found: 757a7d3fba8d43dda36474539b7296ba
I20250905 19:52:35.933123 1170 tablet_bootstrap.cc:654] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:35.933166 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0267ac3e3f1a45978807737eca5d91ef because the tablet was not found. No further retry: Not found: Tablet not found: 0267ac3e3f1a45978807737eca5d91ef
W20250905 19:52:35.933375 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 00d8144bb4f14358af539b7aae326918 because the tablet was not found. No further retry: Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918
W20250905 19:52:35.933544 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 0d4204f61929498188c18529148a8d13 because the tablet was not found. No further retry: Not found: Tablet not found: 0d4204f61929498188c18529148a8d13
W20250905 19:52:35.933715 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 60a41dbd8782460cb0d3c5f8bd59354f because the tablet was not found. No further retry: Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
W20250905 19:52:35.933895 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f because the tablet was not found. No further retry: Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:35.934376 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 1 got 0
I20250905 19:52:35.934710 2652 rpcz_store.cc:276] 0905 19:52:31.622247 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.622819 (+ 572us) service_pool.cc:225] Handling call
0905 19:52:35.880812 (+4257993us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:35.880822 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.887246 (+ 6424us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.914545 (+ 27299us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:35.927115 (+ 12570us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4256240}
I20250905 19:52:35.934767 1532 catalog_manager.cc:5582] T 97d949c3c24e40fe8fefd3a5758d57ea P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:35.935375 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ac5c88cf8d15433a940e937f0a225582 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:35.935709 2652 tablet_replica.cc:331] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:35.936035 2652 raft_consensus.cc:2241] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:52:35.936693 2838 raft_consensus.cc:2802] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:35.936825 2838 raft_consensus.cc:695] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:35.937026 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:35.937059 2838 consensus_queue.cc:237] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.937281 1532 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:35.937953 1536 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet dc95a83ca872495894ef439cdda4e364
I20250905 19:52:35.937800 1522 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d] alter complete (version 2)
I20250905 19:52:35.938658 1536 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 53) took 7832 ms. Trace:
I20250905 19:52:35.938810 1536 rpcz_store.cc:276] 0905 19:52:28.106396 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.325087 (+6218691us) service_pool.cc:225] Handling call
0905 19:52:34.351910 (+ 26823us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:34.352280 (+ 370us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:34.352473 (+ 193us) catalog_manager.cc:2799] Locking tablets
0905 19:52:34.600251 (+247778us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:34.606589 (+ 6338us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:34.606711 (+ 122us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:35.937845 (+1331134us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:35.938601 (+ 756us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:34.601120 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:34.601302 (+ 182us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:34.601312 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:34.601316 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:34.601475 (+ 159us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:34.601495 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:34.601503 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:34.601650 (+ 147us) tablet.cc:699] Row locks acquired
0905 19:52:34.601653 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:34.601701 (+ 48us) write_op.cc:270] Start()
0905 19:52:34.601745 (+ 44us) write_op.cc:276] Timestamp: P: 1757101954601678 usec, L: 0
0905 19:52:34.601750 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:34.602038 (+ 288us) log.cc:844] Serialized 790 byte log entry
0905 19:52:34.605364 (+ 3326us) op_driver.cc:464] REPLICATION: finished
0905 19:52:34.605512 (+ 148us) write_op.cc:301] APPLY: starting
0905 19:52:34.605530 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:34.605590 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:34.605621 (+ 31us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:34.605695 (+ 74us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:34.605726 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:34.605732 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:34.605816 (+ 84us) log.cc:844] Serialized 29 byte log entry
0905 19:52:34.605886 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.605911 (+ 25us) write_op.cc:454] Released schema lock
0905 19:52:34.605920 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:34.606007 (+ 87us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.606013 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":483,"apply.run_wall_time_us":481,"num_ops":2,"prepare.queue_time_us":65,"prepare.run_cpu_time_us":1060,"prepare.run_wall_time_us":1060,"raft.queue_time_us":36,"raft.run_cpu_time_us":324,"raft.run_wall_time_us":325,"replication_time_us":3575}]]}
W20250905 19:52:35.939914 1536 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 72) took 1491 ms (client timeout 1000 ms). Trace:
W20250905 19:52:35.940078 1536 rpcz_store.cc:269] 0905 19:52:34.448204 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.939820 (+1491616us) service_pool.cc:212] Skipping call since client already timed out
0905 19:52:35.939866 (+ 46us) inbound_call.cc:173] Queueing failure response
Metrics: {}
W20250905 19:52:35.945672 1543 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 9f5d19f0216a4f069237e48c62ee7875 consensus state has no leader
I20250905 19:52:35.951236 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:26 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.951551 739 tablet_service.cc:1515] Processing DeleteTablet for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.951810 739 tablet_service.cc:1515] Processing DeleteTablet for tablet f48f9f789b6e46ec871a0fd007770dfc with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.952061 878 tablet_service.cc:1515] Processing DeleteTablet for tablet dc95a83ca872495894ef439cdda4e364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:34 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.952307 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 9db56577a26e407b97fc33158c2c466c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:28 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.952534 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 3602094b96c14e66b997e8035fc53c40 with delete_type TABLET_DATA_DELETED (replaced by tablet 03a12cf719f34096bb82f5846bbe3b2d) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.952757 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.952920 750 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.953100 750 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.953251 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 03a12cf719f34096bb82f5846bbe3b2d with delete_type TABLET_DATA_DELETED (replaced by tablet 03cf686811e041e7a001669b3d86e28a) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.953428 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a8b6823580cb4040b49ed0a49b748965 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.953581 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 00ac56c449ba4b7b8dbd5e87120799d9 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.953747 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 42f3c1efb94e446fb0128b6b412821e2 with delete_type TABLET_DATA_DELETED (replaced by tablet 239539ac19124de192cadae494720a2b) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.953797 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.953873 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 9630f1feb6d74472b5f6f4b54151311c with delete_type TABLET_DATA_DELETED (replaced by tablet 6ee9f4fe3a464abaaca48f22ea183c0e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954036 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 2f581403ec924fb0948400df41ed8aab with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954222 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a5f076d9917f40d899d7422ad974fe0f with delete_type TABLET_DATA_DELETED (replaced by tablet a2ff206366ad4344b4df5fc6a5584f8a) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954440 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 00d8144bb4f14358af539b7aae326918 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954468 746 tablet_service.cc:1515] Processing DeleteTablet for tablet dc95a83ca872495894ef439cdda4e364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:34 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954591 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954674 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 53ac46ce05694bc2b53e195638be9e23 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954770 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 60a41dbd8782460cb0d3c5f8bd59354f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:18 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954838 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954934 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 9193487b4ab74dafa76a386ddcaf674b with delete_type TABLET_DATA_DELETED (replaced by tablet 6b8bbbcac06944469e13f14ab1d80402) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.954981 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 723168b075874dfcb3eb50c5134c499e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955134 750 tablet_service.cc:1515] Processing DeleteTablet for tablet a3805127dd78489e8545da7414ead904 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955179 746 tablet_service.cc:1515] Processing DeleteTablet for tablet 2d84653c9e7a4857a7580fa85f8e4732 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955349 750 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955391 746 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955555 756 tablet_service.cc:1515] Processing DeleteTablet for tablet 24bcd21f19ba465aa550102e0fccb674 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955613 747 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.955812 754 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.956017 748 tablet_service.cc:1515] Processing DeleteTablet for tablet 578a6b9224cc4d8585d6c4b241762e1b with delete_type TABLET_DATA_DELETED (replaced by tablet a3a50526bb604a218649b24cbf65a65e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.956115 755 tablet_service.cc:1515] Processing DeleteTablet for tablet 302053e1611b4e22a9a2e7b60ae68f27 with delete_type TABLET_DATA_DELETED (replaced by tablet ac5c88cf8d15433a940e937f0a225582) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.956215 737 tablet_service.cc:1515] Processing DeleteTablet for tablet 3609de83732d4ad8857372acfbb5c32a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.956437 741 tablet_service.cc:1515] Processing DeleteTablet for tablet a2ff206366ad4344b4df5fc6a5584f8a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.956655 740 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.956837 749 tablet_service.cc:1515] Processing DeleteTablet for tablet a1b4989f17454aa58c05727b797c2009 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.957038 752 tablet_service.cc:1515] Processing DeleteTablet for tablet a8a32f742eaf473e9f6f9bf214f53679 with delete_type TABLET_DATA_DELETED (replaced by tablet 723168b075874dfcb3eb50c5134c499e) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.957234 745 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.957450 738 tablet_service.cc:1515] Processing DeleteTablet for tablet 6221d7d058944bdb9aa730791cf2417b with delete_type TABLET_DATA_DELETED (replaced by tablet 9f5d19f0216a4f069237e48c62ee7875) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.957648 742 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.957849 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 with delete_type TABLET_DATA_DELETED (replaced by tablet ab4f69c45295422f85b310aa55220e9b) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.958036 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 98e5ca7bd3e84916a11e01afbf48171d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:19 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.958226 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.958403 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b13d282e1dc54d1faa2882c9b39d17cb with delete_type TABLET_DATA_DELETED (replaced by tablet c14dc571b586405f8fd1eaa452998978) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.958645 739 tablet_service.cc:1515] Processing DeleteTablet for tablet b0702063ec1c4f6fba58e6d4f3912212 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:17 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.964273 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 091ef981d6b54756830927ea111e78aa with delete_type TABLET_DATA_DELETED (replaced by tablet dc95a83ca872495894ef439cdda4e364) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.964684 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.965098 739 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.965430 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.965775 739 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.965988 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 6f3473785fff45d692bd50b1015ffe39 with delete_type TABLET_DATA_DELETED (replaced by tablet bd9ae8fd5fdc4b539308268ba9729cd6) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.966171 739 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.966224 875 tablet_service.cc:1515] Processing DeleteTablet for tablet c46ea498d6a14d1ebdffea415cfc7114 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.966411 875 tablet_service.cc:1515] Processing DeleteTablet for tablet d296428b70f9411190b6c408d81639f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.966609 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.966848 891 tablet_service.cc:1515] Processing DeleteTablet for tablet fd793bce71e344f18bb177a53ab1bfc1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.966910 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 6d12c7fbb84849c68274810b3fdcba04 with delete_type TABLET_DATA_DELETED (replaced by tablet a5f076d9917f40d899d7422ad974fe0f) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.967005 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 44769f1170354870ae2d34fdb67ffbae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.967140 878 tablet_service.cc:1515] Processing DeleteTablet for tablet ba161282480e4736889559a2e37b195f with delete_type TABLET_DATA_DELETED (replaced by tablet 8faf429d811347a88880e8198daa59cc) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:35.967253 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 43111a3512fb4fc0bdf4ee245b16695b with delete_type TABLET_DATA_DELETED (replaced by tablet efb70327f55844879f89449a7116010f) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:35.968729 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 8503f6918ffb44efa36214633c93e31d with delete_type TABLET_DATA_DELETED (replaced by tablet 8ee9e38dc3574c1ba9190e06557423f6) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:35.969868 1547 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 70) took 3978 ms (client timeout 1000 ms). Trace:
W20250905 19:52:35.970007 1547 rpcz_store.cc:269] 0905 19:52:31.990955 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.765237 (+774282us) service_pool.cc:225] Handling call
0905 19:52:35.969773 (+3204536us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":4721,"reported_tablets":206,"spinlock_wait_cycles":1280}
I20250905 19:52:35.966854 1171 tablet_bootstrap.cc:492] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:35.974565 1171 ts_tablet_manager.cc:1397] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.070s user 0.004s sys 0.000s
W20250905 19:52:35.974985 848 consensus_peers.cc:489] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:35.975432 1171 raft_consensus.cc:357] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.975646 1171 raft_consensus.cc:383] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:35.975706 1171 raft_consensus.cc:738] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:35.975935 1171 consensus_queue.cc:260] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.976588 1171 ts_tablet_manager.cc:1428] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:35.977078 1171 tablet_bootstrap.cc:492] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:35.987319 1522 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 69) took 5095 ms (client timeout 1000 ms). Trace:
W20250905 19:52:35.987459 1522 rpcz_store.cc:269] 0905 19:52:30.891370 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.486967 (+595597us) service_pool.cc:225] Handling call
0905 19:52:32.357847 (+870880us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.357870 (+ 23us) write_op.cc:454] Released schema lock
0905 19:52:32.757855 (+399985us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.757864 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:32.809915 (+ 52051us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.809923 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:35.987173 (+3177250us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:32.346467 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.346610 (+ 143us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.346617 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:32.346622 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:32.346758 (+ 136us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.346777 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.346784 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:32.346873 (+ 89us) tablet.cc:699] Row locks acquired
0905 19:52:32.346877 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:32.346922 (+ 45us) write_op.cc:270] Start()
0905 19:52:32.346967 (+ 45us) write_op.cc:276] Timestamp: P: 1757101952346898 usec, L: 0
0905 19:52:32.346972 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.347217 (+ 245us) log.cc:844] Serialized 356 byte log entry
0905 19:52:32.354752 (+ 7535us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.354902 (+ 150us) write_op.cc:301] APPLY: starting
0905 19:52:32.354921 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.354966 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.355000 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.355065 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.355100 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:32.355109 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:32.355213 (+ 104us) log.cc:844] Serialized 23 byte log entry
0905 19:52:32.355295 (+ 82us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.355318 (+ 23us) write_op.cc:454] Released schema lock
0905 19:52:32.355325 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:32.749092 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.749216 (+ 124us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.749224 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:32.749229 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:32.749357 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.749371 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.749376 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:32.749442 (+ 66us) tablet.cc:699] Row locks acquired
0905 19:52:32.749444 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:32.749487 (+ 43us) write_op.cc:270] Start()
0905 19:52:32.749525 (+ 38us) write_op.cc:276] Timestamp: P: 1757101952749460 usec, L: 0
0905 19:52:32.749531 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.749759 (+ 228us) log.cc:844] Serialized 356 byte log entry
0905 19:52:32.755044 (+ 5285us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.755164 (+ 120us) write_op.cc:301] APPLY: starting
0905 19:52:32.755181 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.755229 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.755255 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.755309 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.755347 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:32.755356 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:32.755435 (+ 79us) log.cc:844] Serialized 23 byte log entry
0905 19:52:32.755503 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.755522 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:32.755530 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:32.803190 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.803316 (+ 126us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.803322 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:32.803324 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:32.803429 (+ 105us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.803442 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.803447 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:32.803508 (+ 61us) tablet.cc:699] Row locks acquired
0905 19:52:32.803512 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:32.803548 (+ 36us) write_op.cc:270] Start()
0905 19:52:32.803579 (+ 31us) write_op.cc:276] Timestamp: P: 1757101952803526 usec, L: 0
0905 19:52:32.803584 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.803772 (+ 188us) log.cc:844] Serialized 356 byte log entry
0905 19:52:32.807317 (+ 3545us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.807436 (+ 119us) write_op.cc:301] APPLY: starting
0905 19:52:32.807450 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.807492 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.807521 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.807575 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.807606 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:32.807614 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:32.807689 (+ 75us) log.cc:844] Serialized 23 byte log entry
0905 19:52:32.807764 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.807783 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:32.807789 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:34.161457 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:34.161606 (+ 149us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:34.161615 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:34.161618 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:34.161767 (+ 149us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:34.161786 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:34.161793 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:34.161902 (+ 109us) tablet.cc:699] Row locks acquired
0905 19:52:34.161906 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:34.161954 (+ 48us) write_op.cc:270] Start()
0905 19:52:34.161996 (+ 42us) write_op.cc:276] Timestamp: P: 1757101954161933 usec, L: 0
0905 19:52:34.162003 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:34.162234 (+ 231us) log.cc:844] Serialized 356 byte log entry
0905 19:52:34.165887 (+ 3653us) op_driver.cc:464] REPLICATION: finished
0905 19:52:34.166006 (+ 119us) write_op.cc:301] APPLY: starting
0905 19:52:34.166023 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:34.166069 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:34.166098 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:34.166156 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:34.166192 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:34.166204 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:34.166292 (+ 88us) log.cc:844] Serialized 23 byte log entry
0905 19:52:34.166366 (+ 74us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.166385 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:34.166391 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:34.166474 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.166478 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":167325,"reported_tablets":208,"spinlock_wait_cycles":768,"child_traces":[["op",{"apply.queue_time_us":93,"apply.run_cpu_time_us":481,"apply.run_wall_time_us":479,"num_ops":1,"prepare.queue_time_us":77,"prepare.run_cpu_time_us":870,"prepare.run_wall_time_us":885,"raft.queue_time_us":481,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":334,"replication_time_us":7737}],["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":425,"apply.run_wall_time_us":425,"num_ops":1,"prepare.queue_time_us":52,"prepare.run_cpu_time_us":787,"prepare.run_wall_time_us":788,"raft.queue_time_us":74,"raft.run_cpu_time_us":235,"raft.run_wall_time_us":234,"replication_time_us":5482}],["op",{"apply.queue_time_us":56,"apply.run_cpu_time_us":409,"apply.run_wall_time_us":409,"mutex_wait_us":10925,"num_ops":1,"prepare.queue_time_us":10972,"prepare.run_cpu_time_us":675,"prepare.run_wall_time_us":674,"raft.queue_time_us":60,"raft.run_cpu_time_us":247,"raft.run_wall_time_us":246,"replication_time_us":3704}],["op",{"apply.queue_time_us":70,"apply.run_cpu_time_us":446,"apply.run_wall_time_us":445,"num_ops":1,"prepare.queue_time_us":86,"prepare.run_cpu_time_us":904,"prepare.run_wall_time_us":904,"raft.queue_time_us":254,"raft.run_cpu_time_us":268,"raft.run_wall_time_us":269,"replication_time_us":3852}]]}
I20250905 19:52:35.992234 2903 raft_consensus.cc:491] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.992398 2903 raft_consensus.cc:513] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.992976 2903 leader_election.cc:290] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:35.993779 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:35.994894 848 leader_election.cc:343] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:35.997838 1849 raft_consensus.cc:491] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:35.998023 1849 raft_consensus.cc:513] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:35.998672 1849 leader_election.cc:290] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.001382 1171 tablet_bootstrap.cc:654] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.007021 1540 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 78) took 3618 ms (client timeout 1000 ms). Trace:
I20250905 19:52:36.007426 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
W20250905 19:52:36.008697 1540 rpcz_store.cc:269] 0905 19:52:32.388369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.834333 (+445964us) service_pool.cc:225] Handling call
0905 19:52:36.006958 (+3172625us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:34.080079 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:34.080247 (+ 168us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:34.080255 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:34.080257 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:34.100556 (+ 20299us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:34.100577 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:34.100584 (+ 7us) tablet.cc:683] Acquiring locks for 4 operations
0905 19:52:34.100773 (+ 189us) tablet.cc:699] Row locks acquired
0905 19:52:34.100777 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:34.100834 (+ 57us) write_op.cc:270] Start()
0905 19:52:34.100883 (+ 49us) write_op.cc:276] Timestamp: P: 1757101954100813 usec, L: 0
0905 19:52:34.100889 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:34.101154 (+ 265us) log.cc:844] Serialized 1650 byte log entry
0905 19:52:34.108780 (+ 7626us) op_driver.cc:464] REPLICATION: finished
0905 19:52:34.108967 (+ 187us) write_op.cc:301] APPLY: starting
0905 19:52:34.108983 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:34.109072 (+ 89us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:34.109108 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:34.109197 (+ 89us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:34.109241 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0905 19:52:34.109249 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:34.109360 (+ 111us) log.cc:844] Serialized 41 byte log entry
0905 19:52:34.109426 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.109457 (+ 31us) write_op.cc:454] Released schema lock
0905 19:52:34.109464 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:34.109549 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:34.109553 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":53679,"reported_tablets":183,"child_traces":[["op",{"apply.queue_time_us":133,"apply.run_cpu_time_us":571,"apply.run_wall_time_us":570,"num_ops":4,"prepare.queue_time_us":2078,"prepare.run_cpu_time_us":1353,"prepare.run_wall_time_us":21201,"raft.queue_time_us":1468,"raft.run_cpu_time_us":217,"raft.run_wall_time_us":218,"replication_time_us":7854}]]}
W20250905 19:52:36.008478 1543 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 77) took 4702 ms (client timeout 1000 ms). Trace:
W20250905 19:52:36.009552 1543 rpcz_store.cc:269] 0905 19:52:31.306181 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.758161 (+451980us) service_pool.cc:225] Handling call
0905 19:52:32.810030 (+1051869us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.810035 (+ 5us) write_op.cc:454] Released schema lock
0905 19:52:36.008439 (+3198404us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:32.793939 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.794085 (+ 146us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.794092 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:32.794094 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:32.794488 (+ 394us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.794507 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.794515 (+ 8us) tablet.cc:683] Acquiring locks for 14 operations
0905 19:52:32.794875 (+ 360us) tablet.cc:699] Row locks acquired
0905 19:52:32.794878 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:32.794921 (+ 43us) write_op.cc:270] Start()
0905 19:52:32.794957 (+ 36us) write_op.cc:276] Timestamp: P: 1757101952794895 usec, L: 0
0905 19:52:32.794961 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.795188 (+ 227us) log.cc:844] Serialized 5390 byte log entry
0905 19:52:32.799813 (+ 4625us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.799984 (+ 171us) write_op.cc:301] APPLY: starting
0905 19:52:32.800003 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.800113 (+ 110us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.800152 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.800335 (+ 183us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.800380 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=14
0905 19:52:32.800390 (+ 10us) write_op.cc:312] APPLY: finished
0905 19:52:32.800505 (+ 115us) log.cc:844] Serialized 101 byte log entry
0905 19:52:32.800569 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.800603 (+ 34us) write_op.cc:454] Released schema lock
0905 19:52:32.800609 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":44925,"reported_tablets":192,"child_traces":[["op",{"apply.queue_time_us":101,"apply.run_cpu_time_us":703,"apply.run_wall_time_us":702,"num_ops":14,"prepare.queue_time_us":6048,"prepare.run_cpu_time_us":1357,"prepare.run_wall_time_us":1357,"raft.queue_time_us":509,"raft.run_cpu_time_us":289,"raft.run_wall_time_us":289,"replication_time_us":4819}]]}
I20250905 19:52:36.010454 1543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_85e1b8aa7bd340459631f0b02ae94b9d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:36.011453 1543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_85e1b8aa7bd340459631f0b02ae94b9d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:36.014387 1542 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 67) took 3978 ms (client timeout 1000 ms). Trace:
W20250905 19:52:36.014552 1542 rpcz_store.cc:269] 0905 19:52:32.036021 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.769308 (+733287us) service_pool.cc:225] Handling call
0905 19:52:36.014345 (+3245037us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:33.605679 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:33.638737 (+ 33058us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:33.638747 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:33.638752 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:33.644122 (+ 5370us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:33.644135 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:33.644141 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:33.644204 (+ 63us) tablet.cc:699] Row locks acquired
0905 19:52:33.644206 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:33.644249 (+ 43us) write_op.cc:270] Start()
0905 19:52:33.644281 (+ 32us) write_op.cc:276] Timestamp: P: 1757101953644230 usec, L: 0
0905 19:52:33.644285 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:33.644509 (+ 224us) log.cc:844] Serialized 356 byte log entry
0905 19:52:33.691442 (+ 46933us) op_driver.cc:464] REPLICATION: finished
0905 19:52:33.779262 (+ 87820us) write_op.cc:301] APPLY: starting
0905 19:52:33.779281 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:33.779329 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:33.779457 (+ 128us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:33.779527 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:33.779571 (+ 44us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:33.779578 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:33.779672 (+ 94us) log.cc:844] Serialized 23 byte log entry
0905 19:52:33.779752 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:33.779773 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:33.779782 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":56874,"reported_tablets":190,"child_traces":[["op",{"apply.queue_time_us":87728,"apply.run_cpu_time_us":555,"apply.run_wall_time_us":608,"num_ops":1,"prepare.queue_time_us":22897,"prepare.run_cpu_time_us":1094,"prepare.run_wall_time_us":39414,"raft.queue_time_us":58,"raft.run_cpu_time_us":343,"raft.run_wall_time_us":554,"replication_time_us":47102,"thread_start_us":156861,"threads_started":2}]]}
I20250905 19:52:36.015550 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 52c99a76802b481c98f77dbd90de7cbb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:36.016162 2379 tablet_replica.cc:331] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.016425 2379 raft_consensus.cc:2241] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.016614 2379 raft_consensus.cc:2270] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.017226 2379 ts_tablet_manager.cc:1905] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.017333 1849 raft_consensus.cc:491] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.017513 1849 raft_consensus.cc:513] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.018292 1849 leader_election.cc:290] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.020848 1171 tablet_bootstrap.cc:492] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.021004 1171 ts_tablet_manager.cc:1397] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.044s user 0.005s sys 0.002s
I20250905 19:52:36.021709 1171 raft_consensus.cc:357] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.021881 1171 raft_consensus.cc:383] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.021940 1171 raft_consensus.cc:738] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.022161 1171 consensus_queue.cc:260] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.022747 1171 ts_tablet_manager.cc:1428] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:36.023062 1171 tablet_bootstrap.cc:492] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.024582 1171 tablet_bootstrap.cc:654] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.025671 1542 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_2cde62a9512e4445b5d0eb12119380f1" } new_table_name: "default.Table_0d6fe65405f543d7a70643f94bf22da5" modify_external_catalogs: true
W20250905 19:52:36.023365 1517 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 66) took 5105 ms (client timeout 1000 ms). Trace:
I20250905 19:52:36.028307 1540 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_a82a3756e48a4bfc99f4aacf9b57f6d3" } new_table_name: "default.Table_fe3dee104cc640359210483bc31532cf" modify_external_catalogs: true
W20250905 19:52:36.029013 1546 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 68) took 2959 ms (client timeout 1000 ms). Trace:
W20250905 19:52:36.029178 1546 rpcz_store.cc:269] 0905 19:52:33.069116 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.002785 (+933669us) service_pool.cc:225] Handling call
0905 19:52:36.028957 (+2026172us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":22915,"reported_tablets":177}
I20250905 19:52:36.029676 1546 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_a3f620901eff44dea30bf9e4de3c5552" } new_table_name: "default.Table_e34bd3d81ce24fffaf905102775bb2e4" modify_external_catalogs: true
W20250905 19:52:36.027784 1517 rpcz_store.cc:269] 0905 19:52:30.918065 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.680969 (+762904us) service_pool.cc:225] Handling call
0905 19:52:31.821524 (+140555us) spinlock_profiling.cc:243] Waited 31.1 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec16f8 00007fabe1cfbc86 00007fabe1cfb9b5 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:32.733847 (+912323us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.733856 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:36.023319 (+3289463us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:32.513709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.513849 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.513856 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:32.513859 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:32.514057 (+ 198us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.514072 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.514078 (+ 6us) tablet.cc:683] Acquiring locks for 4 operations
0905 19:52:32.514193 (+ 115us) tablet.cc:699] Row locks acquired
0905 19:52:32.514195 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:32.514239 (+ 44us) write_op.cc:270] Start()
0905 19:52:32.514284 (+ 45us) write_op.cc:276] Timestamp: P: 1757101952514214 usec, L: 0
0905 19:52:32.514288 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.514489 (+ 201us) log.cc:844] Serialized 1650 byte log entry
0905 19:52:32.517846 (+ 3357us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.517966 (+ 120us) write_op.cc:301] APPLY: starting
0905 19:52:32.517985 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.518052 (+ 67us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.518086 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.518179 (+ 93us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.518218 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=4
0905 19:52:32.518226 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:32.518352 (+ 126us) log.cc:844] Serialized 41 byte log entry
0905 19:52:32.518417 (+ 65us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.518440 (+ 23us) write_op.cc:454] Released schema lock
0905 19:52:32.518447 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:32.518523 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.518527 (+ 4us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:32.599199 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.599337 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.599345 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:32.599347 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:32.599475 (+ 128us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.599491 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.599498 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:32.599573 (+ 75us) tablet.cc:699] Row locks acquired
0905 19:52:32.599575 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:32.599619 (+ 44us) write_op.cc:270] Start()
0905 19:52:32.599656 (+ 37us) write_op.cc:276] Timestamp: P: 1757101952599593 usec, L: 0
0905 19:52:32.599660 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.599890 (+ 230us) log.cc:844] Serialized 356 byte log entry
0905 19:52:32.603857 (+ 3967us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.604009 (+ 152us) write_op.cc:301] APPLY: starting
0905 19:52:32.604025 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.604071 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.604096 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.604153 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.604185 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:32.604196 (+ 11us) write_op.cc:312] APPLY: finished
0905 19:52:32.604303 (+ 107us) log.cc:844] Serialized 23 byte log entry
0905 19:52:32.604366 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.604386 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:32.604393 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:32.604466 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.604473 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:32.715809 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.715921 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.715928 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:32.715933 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:32.716064 (+ 131us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.716076 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.716081 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:32.716138 (+ 57us) tablet.cc:699] Row locks acquired
0905 19:52:32.716140 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:32.716179 (+ 39us) write_op.cc:270] Start()
0905 19:52:32.716213 (+ 34us) write_op.cc:276] Timestamp: P: 1757101952716155 usec, L: 0
0905 19:52:32.716216 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.716397 (+ 181us) log.cc:844] Serialized 356 byte log entry
0905 19:52:32.720145 (+ 3748us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.720272 (+ 127us) write_op.cc:301] APPLY: starting
0905 19:52:32.720290 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.720340 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.720381 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.720438 (+ 57us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.720475 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:32.720483 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:32.720576 (+ 93us) log.cc:844] Serialized 23 byte log entry
0905 19:52:32.720640 (+ 64us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.720660 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:32.720665 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:32.720770 (+ 105us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.720777 (+ 7us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:32.726376 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:32.726495 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:32.726501 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:32.726503 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:32.726611 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:32.726629 (+ 18us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:32.726635 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:32.726702 (+ 67us) tablet.cc:699] Row locks acquired
0905 19:52:32.726704 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:32.726752 (+ 48us) write_op.cc:270] Start()
0905 19:52:32.726789 (+ 37us) write_op.cc:276] Timestamp: P: 1757101952726727 usec, L: 0
0905 19:52:32.726792 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:32.726966 (+ 174us) log.cc:844] Serialized 356 byte log entry
0905 19:52:32.731099 (+ 4133us) op_driver.cc:464] REPLICATION: finished
0905 19:52:32.731199 (+ 100us) write_op.cc:301] APPLY: starting
0905 19:52:32.731216 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:32.731261 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:32.731300 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:32.731363 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:32.731397 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:32.731405 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:32.731487 (+ 82us) log.cc:844] Serialized 23 byte log entry
0905 19:52:32.731546 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:32.731569 (+ 23us) write_op.cc:454] Released schema lock
0905 19:52:32.731577 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"mutex_wait_us":88930,"reported_tablets":188,"spinlock_wait_cycles":71606528,"child_traces":[["op",{"apply.queue_time_us":69,"apply.run_cpu_time_us":548,"apply.run_wall_time_us":548,"num_ops":4,"prepare.queue_time_us":273,"prepare.run_cpu_time_us":889,"prepare.run_wall_time_us":905,"raft.queue_time_us":481,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":253,"replication_time_us":3524}],["op",{"apply.queue_time_us":97,"apply.run_cpu_time_us":448,"apply.run_wall_time_us":448,"num_ops":1,"prepare.queue_time_us":8949,"prepare.run_cpu_time_us":829,"prepare.run_wall_time_us":829,"raft.queue_time_us":432,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":230,"replication_time_us":4160}],["op",{"apply.queue_time_us":72,"apply.run_cpu_time_us":474,"apply.run_wall_time_us":462,"num_ops":1,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":693,"prepare.run_wall_time_us":694,"raft.queue_time_us":402,"raft.run_cpu_time_us":246,"raft.run_wall_time_us":246,"replication_time_us":3895}],["op",{"apply.queue_time_us":55,"apply.run_cpu_time_us":429,"apply.run_wall_time_us":427,"num_ops":1,"prepare.queue_time_us":95,"prepare.run_cpu_time_us":683,"prepare.run_wall_time_us":683,"raft.queue_time_us":59,"raft.run_cpu_time_us":262,"raft.run_wall_time_us":261,"replication_time_us":4272}]]}
I20250905 19:52:36.031088 2376 ts_tablet_manager.cc:1918] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.031930 2376 log.cc:1199] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/37894e39e6f2417fbfcd6d43f1ea2b37
I20250905 19:52:36.032420 2376 ts_tablet_manager.cc:1939] T 37894e39e6f2417fbfcd6d43f1ea2b37 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.032828 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 95) took 6578 ms. Trace:
I20250905 19:52:36.032868 1517 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_6ec4f72266c7447da59b8544cfa98cd6" } new_table_name: "default.Table_631f9bc61e1442c09b7f9ab3a3e881da" modify_external_catalogs: true
I20250905 19:52:36.032970 2376 rpcz_store.cc:276] 0905 19:52:29.454255 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.454308 (+ 53us) service_pool.cc:225] Handling call
0905 19:52:35.891760 (+6437452us) ts_tablet_manager.cc:1145] Deleting tablet 37894e39e6f2417fbfcd6d43f1ea2b37
0905 19:52:35.891769 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:35.900416 (+ 8647us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.030975 (+130559us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.032812 (+ 1837us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6437273}
W20250905 19:52:36.033330 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 148136us
I20250905 19:52:36.033468 2376 tablet_replica.cc:331] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.033653 2376 raft_consensus.cc:2241] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.033803 2376 raft_consensus.cc:2270] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.034353 2376 ts_tablet_manager.cc:1905] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.035427 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 37894e39e6f2417fbfcd6d43f1ea2b37 (table default.Table_f93e2f3cbd884fbf84082156519631cb [id=229a32e449d0470190c8e04a7cbf7a8c]) successfully deleted
W20250905 19:52:36.014394 1539 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:40691 (request call id 79) took 2368 ms (client timeout 1000 ms). Trace:
W20250905 19:52:36.036155 1539 rpcz_store.cc:269] 0905 19:52:33.646011 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.187528 (+541517us) service_pool.cc:225] Handling call
0905 19:52:36.014333 (+1826805us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:35.299871 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.300033 (+ 162us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.300041 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:35.300045 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:35.300368 (+ 323us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.300390 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.300396 (+ 6us) tablet.cc:683] Acquiring locks for 9 operations
0905 19:52:35.302488 (+ 2092us) tablet.cc:699] Row locks acquired
0905 19:52:35.302491 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:35.302554 (+ 63us) write_op.cc:270] Start()
0905 19:52:35.302689 (+ 135us) write_op.cc:276] Timestamp: P: 1757101955302524 usec, L: 0
0905 19:52:35.302696 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:35.302986 (+ 290us) log.cc:844] Serialized 3520 byte log entry
0905 19:52:35.535915 (+232929us) op_driver.cc:464] REPLICATION: finished
0905 19:52:35.567977 (+ 32062us) write_op.cc:301] APPLY: starting
0905 19:52:35.567996 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:35.568089 (+ 93us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:35.568151 (+ 62us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:35.568359 (+ 208us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:35.568405 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0905 19:52:35.568412 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:35.568515 (+ 103us) log.cc:844] Serialized 71 byte log entry
0905 19:52:35.568936 (+ 421us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:35.569005 (+ 69us) write_op.cc:454] Released schema lock
0905 19:52:35.569013 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:35.610728 (+ 41715us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:35.610735 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":29552,"reported_tablets":173,"child_traces":[["op",{"apply.queue_time_us":31935,"apply.run_cpu_time_us":847,"apply.run_wall_time_us":42758,"num_ops":9,"prepare.queue_time_us":97454,"prepare.run_cpu_time_us":1267,"prepare.run_wall_time_us":3984,"raft.queue_time_us":41,"raft.run_cpu_time_us":251,"raft.run_wall_time_us":250,"replication_time_us":233166,"thread_start_us":175112,"threads_started":2}]]}
I20250905 19:52:36.037137 1539 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_b96ef31f0115467fb13cceb26c34bc99" } modify_external_catalogs: true
I20250905 19:52:36.038461 1170 tablet_bootstrap.cc:492] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.039093 1170 ts_tablet_manager.cc:1397] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.114s user 0.003s sys 0.035s
I20250905 19:52:36.040040 1170 raft_consensus.cc:357] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.040273 1170 raft_consensus.cc:383] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.040367 1170 raft_consensus.cc:738] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.040628 1170 consensus_queue.cc:260] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.041127 1170 ts_tablet_manager.cc:1428] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:36.042205 2903 raft_consensus.cc:491] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.042331 2903 raft_consensus.cc:513] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.042840 2903 leader_election.cc:290] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.044148 1171 tablet_bootstrap.cc:492] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.044270 1171 ts_tablet_manager.cc:1397] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.021s user 0.005s sys 0.001s
I20250905 19:52:36.045003 1171 raft_consensus.cc:357] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.045192 1171 raft_consensus.cc:383] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.045243 1171 raft_consensus.cc:738] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.045424 1171 consensus_queue.cc:260] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.045853 1171 ts_tablet_manager.cc:1428] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:36.046361 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6ac13b19fb6427a9ef35a5160033a54" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.048477 1170 tablet_bootstrap.cc:492] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:36.053018 848 leader_election.cc:343] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.058626 1539 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_b96ef31f0115467fb13cceb26c34bc99" } modify_external_catalogs: true
I20250905 19:52:36.063886 1171 tablet_bootstrap.cc:492] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.066227 1171 tablet_bootstrap.cc:654] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.077320 2379 ts_tablet_manager.cc:1918] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250905 19:52:36.077507 2379 log.cc:1199] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8edf4b54974c427a855eee5792d75292
I20250905 19:52:36.077994 2379 ts_tablet_manager.cc:1939] T 8edf4b54974c427a855eee5792d75292 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.078833 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 150) took 4419 ms. Trace:
I20250905 19:52:36.078939 2379 rpcz_store.cc:276] 0905 19:52:31.658908 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.658949 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:36.016091 (+4357142us) ts_tablet_manager.cc:1145] Deleting tablet 8edf4b54974c427a855eee5792d75292
0905 19:52:36.016102 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.049940 (+ 33838us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.077128 (+ 27188us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.078775 (+ 1647us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4356986}
I20250905 19:52:36.079367 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 151) took 4419 ms. Trace:
I20250905 19:52:36.079456 2379 rpcz_store.cc:276] 0905 19:52:31.659612 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.659654 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:36.079207 (+4419553us) ts_tablet_manager.cc:1145] Deleting tablet 29c2b030fafd44ad9e648b2527a4ad07
0905 19:52:36.079217 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.079311 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4419395}
I20250905 19:52:36.079957 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8edf4b54974c427a855eee5792d75292 (table default.Table_93a9ab7d352a4a3da35219aa6f20f432 [id=b5acc86aee9545d99f8d8332c3c76a8e]) successfully deleted
W20250905 19:52:36.080350 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 29c2b030fafd44ad9e648b2527a4ad07 because the tablet was not found. No further retry: Not found: Tablet not found: 29c2b030fafd44ad9e648b2527a4ad07
I20250905 19:52:36.087639 775 raft_consensus.cc:3058] T 7f34bdf651d1412690039dc6ea1095d0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.092998 1849 raft_consensus.cc:491] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.093756 1849 raft_consensus.cc:513] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.094959 2376 ts_tablet_manager.cc:1918] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.095067 2376 log.cc:1199] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:36.096694 2838 raft_consensus.cc:491] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.096853 2838 raft_consensus.cc:513] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.097350 2838 leader_election.cc:290] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.097360 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 152) took 4402 ms. Trace:
I20250905 19:52:36.097759 2376 ts_tablet_manager.cc:1939] T a2ff206366ad4344b4df5fc6a5584f8a P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.098659 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 96) took 6644 ms. Trace:
I20250905 19:52:36.098745 2376 rpcz_store.cc:276] 0905 19:52:29.454359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.454401 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:36.033420 (+6579019us) ts_tablet_manager.cc:1145] Deleting tablet a2ff206366ad4344b4df5fc6a5584f8a
0905 19:52:36.033432 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.070081 (+ 36649us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.094837 (+ 24756us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.098633 (+ 3796us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6578846}
I20250905 19:52:36.099043 2376 tablet_replica.cc:331] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.099215 2376 raft_consensus.cc:2241] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.099360 2376 raft_consensus.cc:2270] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.099758 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a2ff206366ad4344b4df5fc6a5584f8a (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
I20250905 19:52:36.099869 2376 ts_tablet_manager.cc:1905] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.109547 1171 tablet_bootstrap.cc:492] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.109722 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.110149 1849 leader_election.cc:290] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.110834 3020 mvcc.cc:204] Tried to move back new op lower bound from 7197089584675905536 to 7197089567164952576. Current Snapshot: MvccSnapshot[applied={T|T < 7197089584675905536 or (T in {7197089584675905536})}]
I20250905 19:52:36.111222 1170 tablet_bootstrap.cc:654] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.111515 848 leader_election.cc:343] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.112557 2379 rpcz_store.cc:276] 0905 19:52:31.677596 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.677660 (+ 64us) service_pool.cc:225] Handling call
0905 19:52:36.079824 (+4402164us) ts_tablet_manager.cc:1145] Deleting tablet d480531ba94f4b0f9fe40e88fe91512e
0905 19:52:36.079836 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.079895 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4401932}
I20250905 19:52:36.113003 2379 tablet_replica.cc:331] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
W20250905 19:52:36.113180 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d480531ba94f4b0f9fe40e88fe91512e because the tablet was not found. No further retry: Not found: Tablet not found: d480531ba94f4b0f9fe40e88fe91512e
I20250905 19:52:36.113222 2379 raft_consensus.cc:2241] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.113436 2379 raft_consensus.cc:2270] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.114082 2379 ts_tablet_manager.cc:1905] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:35.950942 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet dc95a83ca872495894ef439cdda4e364 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:34 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:35.960970 1023 tablet_service.cc:1515] Processing DeleteTablet for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:24 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:35.960981 1024 tablet_service.cc:1515] Processing DeleteTablet for tablet 5bd9380e8ffc432da4a684934c2df68f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:35.961236 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 1f1a8e7059b348cba34069ef49bf1aad with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:20 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:35.961436 1030 tablet_service.cc:1515] Processing DeleteTablet for tablet a5af8232d31240ff855b79dcea29396b with delete_type TABLET_DATA_DELETED (replaced by tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:35.965049 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet ac5c88cf8d15433a940e937f0a225582 with delete_type TABLET_DATA_DELETED (replaced by tablet ad0760dfee934aacbd4ab1fb81e116c7) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:35.979673 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 with delete_type TABLET_DATA_DELETED (replaced by tablet 160a7814abf9423dbe965c252c20fa7f) from {username='slave'} at 127.0.0.1:39172
W20250905 19:52:35.981427 984 consensus_peers.cc:489] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.115288 1171 ts_tablet_manager.cc:1397] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
W20250905 19:52:36.115298 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 133933us
I20250905 19:52:36.116107 1037 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.116374 1037 raft_consensus.cc:2391] T cd5312852fe345d98039aeea241b7fe8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:35.989498 2496 raft_consensus.cc:491] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.116552 2496 raft_consensus.cc:513] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.117183 2496 leader_election.cc:290] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.002177 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 8edf4b54974c427a855eee5792d75292 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:21 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:36.044848 2998 raft_consensus.cc:491] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.117426 2998 raft_consensus.cc:513] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.117954 2998 leader_election.cc:290] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.012884 2931 consensus_queue.cc:1035] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.074287 1035 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.116629 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.118449 1051 raft_consensus.cc:2391] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:36.065400 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.118630 1044 raft_consensus.cc:2466] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:36.119813 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6fe4620dda4b1296416e01f8ffee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.120169 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ebab2e3698f4f60a770bde9cc019600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.120224 3022 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e38333af29db4bf6af41ee5a25d6701e. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:36.120136 3050 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:36.120455 3022 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:36.120493 3050 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:36.120805 1171 raft_consensus.cc:357] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.120970 1171 raft_consensus.cc:383] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.121021 1171 raft_consensus.cc:738] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.121255 1171 consensus_queue.cc:260] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.121407 1170 tablet_bootstrap.cc:492] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.121419 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6fe4620dda4b1296416e01f8ffee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.121553 1170 ts_tablet_manager.cc:1397] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.073s user 0.002s sys 0.003s
I20250905 19:52:36.121580 914 raft_consensus.cc:2466] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:36.121764 1171 ts_tablet_manager.cc:1428] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250905 19:52:36.122013 1171 tablet_bootstrap.cc:492] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.122288 1170 raft_consensus.cc:357] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.122449 1170 raft_consensus.cc:383] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.122746 986 leader_election.cc:304] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:36.123049 2931 raft_consensus.cc:2802] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.123112 2931 raft_consensus.cc:491] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.123203 2931 raft_consensus.cc:3058] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.123253 1170 raft_consensus.cc:738] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.123492 1170 consensus_queue.cc:260] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.124017 1170 ts_tablet_manager.cc:1428] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:36.124344 1170 tablet_bootstrap.cc:492] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.126546 1170 tablet_bootstrap.cc:654] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.127559 2379 ts_tablet_manager.cc:1918] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.127684 2379 log.cc:1199] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/06850faa25de4b3d83f6bf98ccd370f0
W20250905 19:52:36.128422 984 leader_election.cc:343] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:36.128731 984 leader_election.cc:343] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.132551 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.133642 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ebab2e3698f4f60a770bde9cc019600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.133849 906 raft_consensus.cc:2466] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:36.133931 1171 tablet_bootstrap.cc:654] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.133967 1170 tablet_bootstrap.cc:492] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.133993 2452 rpcz_store.cc:275] Call kudu.consensus.ConsensusService.UpdateConsensus from 127.0.0.1:43940 (request call id 0) took 5899 ms. Trace:
I20250905 19:52:36.134141 1170 ts_tablet_manager.cc:1397] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.002s
I20250905 19:52:36.134302 986 leader_election.cc:304] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.134707 2998 raft_consensus.cc:2802] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.134773 2998 raft_consensus.cc:491] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.134826 2998 raft_consensus.cc:3058] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.134874 1170 raft_consensus.cc:357] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.135258 1170 raft_consensus.cc:383] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.135327 1170 raft_consensus.cc:738] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.135607 1170 consensus_queue.cc:260] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.136157 1170 ts_tablet_manager.cc:1428] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250905 19:52:36.137398 2998 raft_consensus.cc:513] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.138005 2998 leader_election.cc:290] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.138269 1849 raft_consensus.cc:491] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.138396 1849 raft_consensus.cc:513] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.138974 1849 leader_election.cc:290] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.140269 2376 ts_tablet_manager.cc:1918] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.140256 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ebab2e3698f4f60a770bde9cc019600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:36.140379 2376 log.cc:1199] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/fd1280fefc3e43db9928567a734644f1
I20250905 19:52:36.140671 2376 ts_tablet_manager.cc:1939] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:36.140692 984 leader_election.cc:343] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.141270 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 97) took 6686 ms. Trace:
I20250905 19:52:36.141350 2376 rpcz_store.cc:276] 0905 19:52:29.454479 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.454518 (+ 39us) service_pool.cc:225] Handling call
0905 19:52:36.098999 (+6644481us) ts_tablet_manager.cc:1145] Deleting tablet fd1280fefc3e43db9928567a734644f1
0905 19:52:36.099008 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.124389 (+ 25381us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.140132 (+ 15743us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.141249 (+ 1117us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6644359}
I20250905 19:52:36.141647 2376 tablet_replica.cc:331] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:36.141788 711 leader_election.cc:343] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:36.141855 2376 raft_consensus.cc:2241] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.142056 2376 raft_consensus.cc:2270] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.142207 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ebab2e3698f4f60a770bde9cc019600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:36.142349 906 raft_consensus.cc:3058] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.142664 2376 ts_tablet_manager.cc:1905] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:36.144706 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f5d19f0216a4f069237e48c62ee7875 consensus state has no leader
I20250905 19:52:36.145699 1170 tablet_bootstrap.cc:492] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.148983 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet fd1280fefc3e43db9928567a734644f1 (table default.Table_c8afd57df21440b5923aef4dd927ae45 [id=4447588b8d2443e5a561758689326e26]) successfully deleted
I20250905 19:52:36.152084 2379 ts_tablet_manager.cc:1939] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.152585 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 153) took 4472 ms. Trace:
I20250905 19:52:36.152742 2379 rpcz_store.cc:276] 0905 19:52:31.679976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.680024 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:36.112903 (+4432879us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:36.112914 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.121090 (+ 8176us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.127394 (+ 6304us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.152539 (+ 25145us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4432685}
I20250905 19:52:36.153396 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 06850faa25de4b3d83f6bf98ccd370f0 (table default.Table_4ec0491dbe8841049cceaeee7c721211 [id=23b4c9dd0ca7459fa0d0a8b5f7033fd9]) successfully deleted
I20250905 19:52:36.155622 2838 raft_consensus.cc:491] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.158113 2838 raft_consensus.cc:513] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.156468 1171 tablet_bootstrap.cc:492] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.159833 1171 ts_tablet_manager.cc:1397] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.038s user 0.003s sys 0.001s
I20250905 19:52:36.159541 906 raft_consensus.cc:2466] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:36.160398 986 leader_election.cc:304] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.160691 1171 raft_consensus.cc:357] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.160847 1171 raft_consensus.cc:383] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.160867 2998 raft_consensus.cc:2802] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:36.160903 1171 raft_consensus.cc:738] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.161098 2998 raft_consensus.cc:695] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:36.161201 1171 consensus_queue.cc:260] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.161330 2998 consensus_queue.cc:237] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.161991 1171 ts_tablet_manager.cc:1428] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:36.162832 1171 tablet_bootstrap.cc:492] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.163719 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 154) took 4481 ms. Trace:
I20250905 19:52:36.163852 2379 rpcz_store.cc:276] 0905 19:52:31.682560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.682622 (+ 62us) service_pool.cc:225] Handling call
0905 19:52:36.163577 (+4480955us) ts_tablet_manager.cc:1145] Deleting tablet 19e4edddafcd4d22a0e0ae66b0b953e0
0905 19:52:36.163590 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.163679 (+ 89us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4480716}
W20250905 19:52:36.164355 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 19e4edddafcd4d22a0e0ae66b0b953e0 because the tablet was not found. No further retry: Not found: Tablet not found: 19e4edddafcd4d22a0e0ae66b0b953e0
I20250905 19:52:36.165220 1170 tablet_bootstrap.cc:654] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.165247 1171 tablet_bootstrap.cc:654] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.134088 2452 rpcz_store.cc:276] 0905 19:52:30.234292 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:30.234355 (+ 63us) service_pool.cc:225] Handling call
0905 19:52:35.876445 (+5642090us) raft_consensus.cc:1469] Updating replica for 43 ops
0905 19:52:35.903379 (+ 26934us) raft_consensus.cc:1532] Early marking committed up to index 593
0905 19:52:35.923762 (+ 20383us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:35.923767 (+ 5us) write_op.cc:454] Released schema lock
0905 19:52:35.924025 (+ 258us) raft_consensus.cc:1537] Triggering prepare for 43 ops
0905 19:52:35.924910 (+ 885us) write_op.cc:270] Start()
0905 19:52:35.924966 (+ 56us) write_op.cc:276] Timestamp: P: 1757101946103010 usec, L: 0
0905 19:52:35.925859 (+ 893us) write_op.cc:270] Start()
0905 19:52:35.925901 (+ 42us) write_op.cc:276] Timestamp: P: 1757101946104035 usec, L: 0
0905 19:52:35.926568 (+ 667us) write_op.cc:270] Start()
0905 19:52:35.926591 (+ 23us) write_op.cc:276] Timestamp: P: 1757101946139578 usec, L: 0
0905 19:52:35.927068 (+ 477us) write_op.cc:270] Start()
0905 19:52:35.927089 (+ 21us) write_op.cc:276] Timestamp: P: 1757101946253346 usec, L: 0
0905 19:52:35.927598 (+ 509us) write_op.cc:270] Start()
0905 19:52:35.927625 (+ 27us) write_op.cc:276] Timestamp: P: 1757101946254338 usec, L: 0
0905 19:52:35.928168 (+ 543us) write_op.cc:270] Start()
0905 19:52:35.928190 (+ 22us) write_op.cc:276] Timestamp: P: 1757101946255158 usec, L: 0
0905 19:52:35.928674 (+ 484us) write_op.cc:270] Start()
0905 19:52:35.928695 (+ 21us) write_op.cc:276] Timestamp: P: 1757101946255941 usec, L: 0
0905 19:52:35.929191 (+ 496us) write_op.cc:270] Start()
0905 19:52:35.929212 (+ 21us) write_op.cc:276] Timestamp: P: 1757101946301324 usec, L: 0
0905 19:52:35.929704 (+ 492us) write_op.cc:270] Start()
0905 19:52:35.929724 (+ 20us) write_op.cc:276] Timestamp: P: 1757101946312589 usec, L: 0
0905 19:52:35.930147 (+ 423us) write_op.cc:270] Start()
0905 19:52:35.930167 (+ 20us) write_op.cc:276] Timestamp: P: 1757101946313626 usec, L: 0
0905 19:52:35.930682 (+ 515us) write_op.cc:270] Start()
0905 19:52:35.930705 (+ 23us) write_op.cc:276] Timestamp: P: 1757101946314552 usec, L: 0
0905 19:52:35.931166 (+ 461us) write_op.cc:270] Start()
0905 19:52:35.931188 (+ 22us) write_op.cc:276] Timestamp: P: 1757101946315617 usec, L: 0
0905 19:52:35.931790 (+ 602us) write_op.cc:270] Start()
0905 19:52:35.931822 (+ 32us) write_op.cc:276] Timestamp: P: 1757101946373444 usec, L: 0
0905 19:52:35.932406 (+ 584us) write_op.cc:270] Start()
0905 19:52:35.932430 (+ 24us) write_op.cc:276] Timestamp: P: 1757101946379273 usec, L: 0
0905 19:52:35.932926 (+ 496us) write_op.cc:270] Start()
0905 19:52:35.932966 (+ 40us) write_op.cc:276] Timestamp: P: 1757101946389165 usec, L: 0
0905 19:52:35.936715 (+ 3749us) write_op.cc:270] Start()
0905 19:52:35.936752 (+ 37us) write_op.cc:276] Timestamp: P: 1757101946424020 usec, L: 0
0905 19:52:35.937424 (+ 672us) write_op.cc:270] Start()
0905 19:52:35.937453 (+ 29us) write_op.cc:276] Timestamp: P: 1757101946447261 usec, L: 0
0905 19:52:35.978433 (+ 40980us) write_op.cc:270] Start()
0905 19:52:35.978635 (+ 202us) write_op.cc:276] Timestamp: P: 1757101946458307 usec, L: 0
0905 19:52:35.981440 (+ 2805us) write_op.cc:270] Start()
0905 19:52:35.981482 (+ 42us) write_op.cc:276] Timestamp: P: 1757101946470431 usec, L: 0
0905 19:52:35.986939 (+ 5457us) write_op.cc:270] Start()
0905 19:52:35.986982 (+ 43us) write_op.cc:276] Timestamp: P: 1757101946473185 usec, L: 0
0905 19:52:35.987178 (+ 196us) write_op.cc:270] Start()
0905 19:52:35.987204 (+ 26us) write_op.cc:276] Timestamp: P: 1757101946515543 usec, L: 0
0905 19:52:35.987367 (+ 163us) write_op.cc:270] Start()
0905 19:52:35.987391 (+ 24us) write_op.cc:276] Timestamp: P: 1757101947171512 usec, L: 0
0905 19:52:35.987544 (+ 153us) write_op.cc:270] Start()
0905 19:52:35.987604 (+ 60us) write_op.cc:276] Timestamp: P: 1757101947574269 usec, L: 0
0905 19:52:35.987752 (+ 148us) write_op.cc:270] Start()
0905 19:52:35.987776 (+ 24us) write_op.cc:276] Timestamp: P: 1757101947934446 usec, L: 0
0905 19:52:35.987917 (+ 141us) write_op.cc:270] Start()
0905 19:52:35.987939 (+ 22us) write_op.cc:276] Timestamp: P: 1757101948008628 usec, L: 0
0905 19:52:35.988178 (+ 239us) write_op.cc:270] Start()
0905 19:52:35.988206 (+ 28us) write_op.cc:276] Timestamp: P: 1757101948162564 usec, L: 0
0905 19:52:35.988422 (+ 216us) write_op.cc:270] Start()
0905 19:52:35.988444 (+ 22us) write_op.cc:276] Timestamp: P: 1757101948169078 usec, L: 0
0905 19:52:35.988627 (+ 183us) write_op.cc:270] Start()
0905 19:52:35.988653 (+ 26us) write_op.cc:276] Timestamp: P: 1757101948349669 usec, L: 0
0905 19:52:35.988828 (+ 175us) write_op.cc:270] Start()
0905 19:52:35.988851 (+ 23us) write_op.cc:276] Timestamp: P: 1757101948364393 usec, L: 0
0905 19:52:35.989002 (+ 151us) write_op.cc:270] Start()
0905 19:52:35.989024 (+ 22us) write_op.cc:276] Timestamp: P: 1757101948922247 usec, L: 0
0905 19:52:35.989177 (+ 153us) write_op.cc:270] Start()
0905 19:52:35.989198 (+ 21us) write_op.cc:276] Timestamp: P: 1757101949363792 usec, L: 0
0905 19:52:35.989344 (+ 146us) write_op.cc:270] Start()
0905 19:52:35.989368 (+ 24us) write_op.cc:276] Timestamp: P: 1757101949383766 usec, L: 0
0905 19:52:35.989529 (+ 161us) write_op.cc:270] Start()
0905 19:52:35.989550 (+ 21us) write_op.cc:276] Timestamp: P: 1757101949397269 usec, L: 0
0905 19:52:35.989698 (+ 148us) write_op.cc:270] Start()
0905 19:52:35.989720 (+ 22us) write_op.cc:276] Timestamp: P: 1757101949408721 usec, L: 0
0905 19:52:35.989862 (+ 142us) write_op.cc:270] Start()
0905 19:52:35.989885 (+ 23us) write_op.cc:276] Timestamp: P: 1757101949428723 usec, L: 0
0905 19:52:35.990052 (+ 167us) write_op.cc:270] Start()
0905 19:52:35.990075 (+ 23us) write_op.cc:276] Timestamp: P: 1757101949458134 usec, L: 0
0905 19:52:35.990226 (+ 151us) write_op.cc:270] Start()
0905 19:52:35.990263 (+ 37us) write_op.cc:276] Timestamp: P: 1757101949466910 usec, L: 0
0905 19:52:35.990486 (+ 223us) write_op.cc:270] Start()
0905 19:52:35.990509 (+ 23us) write_op.cc:276] Timestamp: P: 1757101949510426 usec, L: 0
0905 19:52:35.990659 (+ 150us) write_op.cc:270] Start()
0905 19:52:35.990682 (+ 23us) write_op.cc:276] Timestamp: P: 1757101949518216 usec, L: 0
0905 19:52:35.990836 (+ 154us) write_op.cc:270] Start()
0905 19:52:35.990858 (+ 22us) write_op.cc:276] Timestamp: P: 1757101949540683 usec, L: 0
0905 19:52:35.991001 (+ 143us) write_op.cc:270] Start()
0905 19:52:35.991022 (+ 21us) write_op.cc:276] Timestamp: P: 1757101949554057 usec, L: 0
0905 19:52:35.991163 (+ 141us) write_op.cc:270] Start()
0905 19:52:35.991186 (+ 23us) write_op.cc:276] Timestamp: P: 1757101949627260 usec, L: 0
0905 19:52:36.010550 (+ 19364us) log.cc:844] Serialized 31804 byte log entry
0905 19:52:36.019265 (+ 8715us) raft_consensus.cc:1643] Marking committed up to 636
0905 19:52:36.019483 (+ 218us) log.cc:844] Serialized 15 byte log entry
0905 19:52:36.057916 (+ 38433us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.057927 (+ 11us) write_op.cc:454] Released schema lock
0905 19:52:36.061625 (+ 3698us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.061633 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:36.062808 (+ 1175us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.062814 (+ 6us) write_op.cc:454] Released schema lock
0905 19:52:36.063611 (+ 797us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.063619 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:36.069407 (+ 5788us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.069416 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:36.070473 (+ 1057us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.070480 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.071574 (+ 1094us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.071581 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.072425 (+ 844us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.072431 (+ 6us) write_op.cc:454] Released schema lock
0905 19:52:36.073241 (+ 810us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.073248 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.075891 (+ 2643us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.075898 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.076724 (+ 826us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.076730 (+ 6us) write_op.cc:454] Released schema lock
0905 19:52:36.077582 (+ 852us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.077589 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.078253 (+ 664us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.078260 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.080109 (+ 1849us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.080118 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:36.080848 (+ 730us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.080851 (+ 3us) write_op.cc:454] Released schema lock
0905 19:52:36.081602 (+ 751us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.081608 (+ 6us) write_op.cc:454] Released schema lock
0905 19:52:36.090153 (+ 8545us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.090160 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.092362 (+ 2202us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.092370 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:36.094646 (+ 2276us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.094653 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.097461 (+ 2808us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.097469 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:36.099476 (+ 2007us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.099483 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.100571 (+ 1088us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.100578 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.105369 (+ 4791us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.105374 (+ 5us) write_op.cc:454] Released schema lock
0905 19:52:36.106778 (+ 1404us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.106783 (+ 5us) write_op.cc:454] Released schema lock
0905 19:52:36.107693 (+ 910us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.107700 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.111070 (+ 3370us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.111078 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:36.112749 (+ 1671us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.112756 (+ 7us) write_op.cc:454] Released schema lock
0905 19:52:36.113648 (+ 892us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.113652 (+ 4us) write_op.cc:454] Released schema lock
0905 19:52:36.114974 (+ 1322us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.114978 (+ 4us) write_op.cc:454] Released schema lock
0905 19:52:36.116303 (+ 1325us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.116311 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:36.117581 (+ 1270us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.117585 (+ 4us) write_op.cc:454] Released schema lock
0905 19:52:36.118974 (+ 1389us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.118978 (+ 4us) write_op.cc:454] Released schema lock
0905 19:52:36.119036 (+ 58us) raft_consensus.cc:1695] Filling consensus response to leader.
0905 19:52:36.120072 (+ 1036us) spinlock_profiling.cc:243] Waited 64 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1de1ee5 00007f73d1df5d45 00007f73d1df5af1 00007f73cb3aa32d 00007f73ca07a74e 00007f73ca07a49d 0000000000a774dc 00007f73be5ea14f
0905 19:52:36.120161 (+ 89us) spinlock_profiling.cc:243] Waited 213 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1de1ee5 00007f73d1df5d45 00007f73d1df5af1 00007f73cb3aa32d 00007f73ca07a74e 00007f73ca07a49d 0000000000a774dc 00007f73be5ea14f
0905 19:52:36.133690 (+ 13529us) spinlock_profiling.cc:243] Waited 19.6 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f73ca04bd59 00007f73ca047b07 00007f73cddae36d 00007f73c8266346 00007f73c8266075 00007f73c538d0be 00007f73c538bf9e 00007f73c53921d5 00007f73c5392aff 00007f73c5392928
0905 19:52:36.133702 (+ 12us) raft_consensus.cc:1666] Waiting on the replicates to finish logging
0905 19:52:36.133742 (+ 40us) raft_consensus.cc:1683] finished
0905 19:52:36.133746 (+ 4us) raft_consensus.cc:1689] UpdateReplicas() finished
0905 19:52:36.133925 (+ 179us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:35.925104 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.925234 (+ 130us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.925240 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:35.925242 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:35.925368 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.925382 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.925386 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.925458 (+ 72us) tablet.cc:699] Row locks acquired
0905 19:52:35.925460 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.055462 (+130002us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.055566 (+ 104us) write_op.cc:301] APPLY: starting
0905 19:52:36.055585 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.055634 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.055637 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.055777 (+ 140us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.055876 (+ 99us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.055903 (+ 27us) write_op.cc:312] APPLY: finished
0905 19:52:36.056027 (+ 124us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.056106 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.056134 (+ 28us) write_op.cc:454] Released schema lock
0905 19:52:36.056145 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.926053 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.926192 (+ 139us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.926200 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:35.926204 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:35.926319 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.926330 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.926335 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.926413 (+ 78us) tablet.cc:699] Row locks acquired
0905 19:52:35.926415 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.058060 (+131645us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.058168 (+ 108us) write_op.cc:301] APPLY: starting
0905 19:52:36.058185 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.058258 (+ 73us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.058263 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.058417 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.058498 (+ 81us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.058531 (+ 33us) write_op.cc:312] APPLY: finished
0905 19:52:36.058625 (+ 94us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.058706 (+ 81us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.058735 (+ 29us) write_op.cc:454] Released schema lock
0905 19:52:36.058744 (+ 9us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.926678 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.926756 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.926760 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:35.926763 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:35.926833 (+ 70us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.926843 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.926847 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.926905 (+ 58us) tablet.cc:699] Row locks acquired
0905 19:52:35.926908 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:36.061737 (+134829us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.061834 (+ 97us) write_op.cc:301] APPLY: starting
0905 19:52:36.061850 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.061902 (+ 52us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.061907 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.062046 (+ 139us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.062127 (+ 81us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.062168 (+ 41us) write_op.cc:312] APPLY: finished
0905 19:52:36.062251 (+ 83us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.062341 (+ 90us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.062360 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:36.062368 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.927175 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.927250 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.927255 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:35.927257 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:35.927338 (+ 81us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.927348 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.927352 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.927401 (+ 49us) tablet.cc:699] Row locks acquired
0905 19:52:35.927404 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:36.062901 (+135497us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.062988 (+ 87us) write_op.cc:301] APPLY: starting
0905 19:52:36.063002 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.063043 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.063047 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.063132 (+ 85us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.063167 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.063193 (+ 26us) write_op.cc:312] APPLY: finished
0905 19:52:36.063249 (+ 56us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.063328 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.063346 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:36.063353 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.927720 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.927813 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.927818 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:35.927821 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:35.927881 (+ 60us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.927891 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.927895 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.927946 (+ 51us) tablet.cc:699] Row locks acquired
0905 19:52:35.927948 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.063715 (+135767us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.067134 (+ 3419us) write_op.cc:301] APPLY: starting
0905 19:52:36.067149 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.067189 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.067191 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.067317 (+ 126us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.067347 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:36.067382 (+ 35us) write_op.cc:312] APPLY: finished
0905 19:52:36.067449 (+ 67us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.067508 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.067523 (+ 15us) write_op.cc:454] Released schema lock
0905 19:52:36.067529 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.928308 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.928386 (+ 78us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.928391 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:35.928394 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:35.928453 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.928463 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.928467 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.928518 (+ 51us) tablet.cc:699] Row locks acquired
0905 19:52:35.928521 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:36.069525 (+141004us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.069622 (+ 97us) write_op.cc:301] APPLY: starting
0905 19:52:36.069640 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.069689 (+ 49us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.069693 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.069913 (+ 220us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.069956 (+ 43us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:36.069996 (+ 40us) write_op.cc:312] APPLY: finished
0905 19:52:36.070080 (+ 84us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.070152 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.070170 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:36.070178 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.928814 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.928897 (+ 83us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.928901 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:35.928903 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:35.928962 (+ 59us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.928972 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.928977 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.929028 (+ 51us) tablet.cc:699] Row locks acquired
0905 19:52:35.929030 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.070561 (+141531us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.070669 (+ 108us) write_op.cc:301] APPLY: starting
0905 19:52:36.070683 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.070734 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.070738 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.070901 (+ 163us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.070939 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:36.070984 (+ 45us) write_op.cc:312] APPLY: finished
0905 19:52:36.071052 (+ 68us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.071298 (+ 246us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.071317 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:36.071325 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.929308 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.929382 (+ 74us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.929386 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:35.929389 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:35.929446 (+ 57us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.929455 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.929460 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.929529 (+ 69us) tablet.cc:699] Row locks acquired
0905 19:52:35.929531 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.071659 (+142128us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.071742 (+ 83us) write_op.cc:301] APPLY: starting
0905 19:52:36.071755 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.071796 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.071800 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.071907 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.071945 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:36.071997 (+ 52us) write_op.cc:312] APPLY: finished
0905 19:52:36.072065 (+ 68us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.072310 (+ 245us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.072328 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:36.072336 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.929810 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.929885 (+ 75us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.929890 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:35.929892 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:35.929950 (+ 58us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.929960 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.929965 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:35.930014 (+ 49us) tablet.cc:699] Row locks acquired
0905 19:52:35.930016 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.072506 (+142490us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.072586 (+ 80us) write_op.cc:301] APPLY: starting
0905 19:52:36.072599 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.072639 (+ 40us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.072644 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.072751 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.072786 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:36.072836 (+ 50us) write_op.cc:312] APPLY: finished
0905 19:52:36.072901 (+ 65us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.073132 (+ 231us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.073149 (+ 17us) write_op.cc:454] Released schema lock
0905 19:52:36.073157 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:35.930249 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:35.930333 (+ 84us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:35.930337 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:35.930339 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:35.930417 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:35.930427 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:35.9
I20250905 19:52:36.168174 2838 leader_election.cc:290] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.168795 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 155) took 4477 ms. Trace:
I20250905 19:52:36.168962 2379 rpcz_store.cc:276] 0905 19:52:31.690765 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.690818 (+ 53us) service_pool.cc:225] Handling call
0905 19:52:36.167866 (+4477048us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:36.167883 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.168048 (+ 165us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4475797}
W20250905 19:52:36.169661 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6ebab2e3698f4f60a770bde9cc019600 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6ebab2e3698f4f60a770bde9cc019600 already in progress: creating tablet
I20250905 19:52:36.170713 771 raft_consensus.cc:3058] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.171635 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 156) took 4475 ms. Trace:
I20250905 19:52:36.171749 2379 rpcz_store.cc:276] 0905 19:52:31.695799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.695850 (+ 51us) service_pool.cc:225] Handling call
0905 19:52:36.171512 (+4475662us) ts_tablet_manager.cc:1145] Deleting tablet a5af8232d31240ff855b79dcea29396b
0905 19:52:36.171523 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.171592 (+ 69us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4475436}
I20250905 19:52:36.172891 2379 tablet_replica.cc:331] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.173149 2379 raft_consensus.cc:2241] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:36.173214 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a5af8232d31240ff855b79dcea29396b because the tablet was not found. No further retry: Not found: Tablet not found: a5af8232d31240ff855b79dcea29396b
I20250905 19:52:36.173343 2379 raft_consensus.cc:2270] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.175822 2838 raft_consensus.cc:491] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.176002 2838 raft_consensus.cc:513] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.176587 2838 leader_election.cc:290] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.180068 2376 ts_tablet_manager.cc:1918] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.180220 2376 log.cc:1199] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/3c824f5591044c4198a51a25661f71e5
I20250905 19:52:36.180614 2376 ts_tablet_manager.cc:1939] T 3c824f5591044c4198a51a25661f71e5 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.181380 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 98) took 6726 ms. Trace:
I20250905 19:52:36.181474 2376 rpcz_store.cc:276] 0905 19:52:29.454610 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.454697 (+ 87us) service_pool.cc:225] Handling call
0905 19:52:36.141602 (+6686905us) ts_tablet_manager.cc:1145] Deleting tablet 3c824f5591044c4198a51a25661f71e5
0905 19:52:36.141612 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.157206 (+ 15594us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.175876 (+ 18670us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.181358 (+ 5482us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6686728}
I20250905 19:52:36.181813 2376 tablet_replica.cc:331] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.182014 2376 raft_consensus.cc:2241] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.181985 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 3c824f5591044c4198a51a25661f71e5 (table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]) successfully deleted
I20250905 19:52:36.182148 2376 raft_consensus.cc:2270] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.184630 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c77dd068ea5c495ca8aa0e7a6823d792" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.185981 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10b4e82c4bc429b9a42db39c03a5267" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.186996 2379 ts_tablet_manager.cc:1905] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:36.187695 848 leader_election.cc:343] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:36.187903 848 leader_election.cc:343] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.188421 2838 raft_consensus.cc:491] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.188583 2838 raft_consensus.cc:513] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.189334 2838 leader_election.cc:290] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.192673 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.194216 2652 pending_rounds.cc:70] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:36.194335 2652 pending_rounds.cc:77] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 2 index: 2 } timestamp: 7197089596070793216 op_type: NO_OP noop_request { }
I20250905 19:52:36.194509 2652 raft_consensus.cc:2887] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:36.195250 2652 raft_consensus.cc:2270] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.196508 2376 ts_tablet_manager.cc:1905] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.198118 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fd1280fefc3e43db9928567a734644f1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.198345 1046 raft_consensus.cc:2466] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
W20250905 19:52:36.199522 848 leader_election.cc:343] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.206826 2379 ts_tablet_manager.cc:1918] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.207017 2379 log.cc:1199] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6bb79e3126074736a23b128a8e14fab6
I20250905 19:52:36.207434 2379 ts_tablet_manager.cc:1939] T 6bb79e3126074736a23b128a8e14fab6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.207851 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 157) took 4483 ms. Trace:
I20250905 19:52:36.211557 1169 tablet_bootstrap.cc:492] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:36.211764 1169 ts_tablet_manager.cc:1397] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.315s user 0.002s sys 0.007s
I20250905 19:52:36.212633 1169 raft_consensus.cc:357] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.212922 1169 raft_consensus.cc:383] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.213017 1169 raft_consensus.cc:738] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.213277 1169 consensus_queue.cc:260] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.213829 1169 ts_tablet_manager.cc:1428] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:36.214130 1169 tablet_bootstrap.cc:492] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:36.216881 2379 rpcz_store.cc:276] 0905 19:52:31.724320 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.724454 (+ 134us) service_pool.cc:225] Handling call
0905 19:52:36.172797 (+4448343us) ts_tablet_manager.cc:1145] Deleting tablet 6bb79e3126074736a23b128a8e14fab6
0905 19:52:36.172810 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.190981 (+ 18171us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.206670 (+ 15689us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.207782 (+ 1112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4447559}
I20250905 19:52:36.217324 2379 tablet_replica.cc:331] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.217523 2379 raft_consensus.cc:2241] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.217697 2379 raft_consensus.cc:2270] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.218250 2379 ts_tablet_manager.cc:1905] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.219138 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6bb79e3126074736a23b128a8e14fab6 (table default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]) successfully deleted
I20250905 19:52:36.228902 775 raft_consensus.cc:3058] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.229491 3063 raft_consensus.cc:491] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:36.229638 3063 raft_consensus.cc:513] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.230436 769 raft_consensus.cc:3058] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.230832 3063 leader_election.cc:290] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.240010 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3de936d2309a411eabbd59f7b73ce7f3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.240952 2838 raft_consensus.cc:491] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.241163 2838 raft_consensus.cc:513] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.247550 2379 ts_tablet_manager.cc:1918] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.247707 2379 log.cc:1199] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:36.248077 2379 ts_tablet_manager.cc:1939] T 2350bd08f977424ba94ee8fa01f66efa P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.248507 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 158) took 4507 ms. Trace:
I20250905 19:52:36.248620 2379 rpcz_store.cc:276] 0905 19:52:31.740627 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.740676 (+ 49us) service_pool.cc:225] Handling call
0905 19:52:36.217269 (+4476593us) ts_tablet_manager.cc:1145] Deleting tablet 2350bd08f977424ba94ee8fa01f66efa
0905 19:52:36.217281 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.232458 (+ 15177us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.247391 (+ 14933us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.248476 (+ 1085us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4476403}
I20250905 19:52:36.249105 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 165) took 3427 ms. Trace:
I20250905 19:52:36.249202 2379 rpcz_store.cc:276] 0905 19:52:32.821429 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.821481 (+ 52us) service_pool.cc:225] Handling call
0905 19:52:36.248986 (+3427505us) ts_tablet_manager.cc:1145] Deleting tablet 01c20eea1b024d7f93f18f0c089efc42
0905 19:52:36.248997 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.249089 (+ 92us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3427267}
I20250905 19:52:36.249570 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 168) took 3427 ms. Trace:
I20250905 19:52:36.249655 2379 rpcz_store.cc:276] 0905 19:52:32.821739 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.821779 (+ 40us) service_pool.cc:225] Handling call
0905 19:52:36.249487 (+3427708us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:36.249497 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.249555 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3427498}
I20250905 19:52:36.249974 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 159) took 3429 ms. Trace:
I20250905 19:52:36.250043 2379 rpcz_store.cc:276] 0905 19:52:32.820747 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.822956 (+ 2209us) service_pool.cc:225] Handling call
0905 19:52:36.249895 (+3426939us) ts_tablet_manager.cc:1145] Deleting tablet 78076f6534814939ae8c2db9a4bd1879
0905 19:52:36.249906 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.249960 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3426762}
I20250905 19:52:36.257869 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca646f7c162e41469f283657e72823b4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.264192 2376 ts_tablet_manager.cc:1918] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.264338 2376 log.cc:1199] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/421b28d047124222ae9cbc603ec22cee
I20250905 19:52:36.264693 2376 ts_tablet_manager.cc:1939] T 421b28d047124222ae9cbc603ec22cee P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.195935 2652 ts_tablet_manager.cc:1905] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.268368 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 160) took 3447 ms. Trace:
I20250905 19:52:36.268479 2379 rpcz_store.cc:276] 0905 19:52:32.820876 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.823168 (+ 2292us) service_pool.cc:225] Handling call
0905 19:52:36.250310 (+3427142us) ts_tablet_manager.cc:1145] Deleting tablet ab4f69c45295422f85b310aa55220e9b
0905 19:52:36.261434 (+ 11124us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.268344 (+ 6910us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3426971}
I20250905 19:52:36.270160 2422 consensus_queue.cc:1035] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.270808 2738 raft_consensus.cc:491] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.270927 2738 raft_consensus.cc:513] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.268883 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 161) took 3447 ms. Trace:
I20250905 19:52:36.271154 2379 rpcz_store.cc:276] 0905 19:52:32.820962 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.823363 (+ 2401us) service_pool.cc:225] Handling call
0905 19:52:36.268766 (+3445403us) ts_tablet_manager.cc:1145] Deleting tablet 421b28d047124222ae9cbc603ec22cee
0905 19:52:36.268777 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.268868 (+ 91us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3445244}
I20250905 19:52:36.271435 2738 leader_election.cc:290] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.271555 2738 raft_consensus.cc:491] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.271677 2738 raft_consensus.cc:513] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.272307 2738 leader_election.cc:290] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:36.273535 986 consensus_peers.cc:489] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:36.274204 848 leader_election.cc:343] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: a1b4989f17454aa58c05727b797c2009
I20250905 19:52:36.280949 2838 leader_election.cc:290] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.271510 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 162) took 3450 ms. Trace:
I20250905 19:52:36.281076 2379 rpcz_store.cc:276] 0905 19:52:32.821112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.824026 (+ 2914us) service_pool.cc:225] Handling call
0905 19:52:36.271430 (+3447404us) ts_tablet_manager.cc:1145] Deleting tablet 2683a36998674f739b41d2a57339954c
0905 19:52:36.271439 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.271494 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3447225}
I20250905 19:52:36.281590 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 163) took 3460 ms. Trace:
I20250905 19:52:36.281673 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9d9815864aa7bf5ca9a5c4485390" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.281975 771 raft_consensus.cc:2466] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:36.284974 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:36.281705 2379 rpcz_store.cc:276] 0905 19:52:32.821215 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.824236 (+ 3021us) service_pool.cc:225] Handling call
0905 19:52:36.281471 (+3457235us) ts_tablet_manager.cc:1145] Deleting tablet f28e403dc2f6471ebea1805fd843cfb1
0905 19:52:36.281484 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.281570 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3457106}
I20250905 19:52:36.285750 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 164) took 3464 ms. Trace:
I20250905 19:52:36.286891 2379 rpcz_store.cc:276] 0905 19:52:32.821316 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.824383 (+ 3067us) service_pool.cc:225] Handling call
0905 19:52:36.285662 (+3461279us) ts_tablet_manager.cc:1145] Deleting tablet 05ae83eea5f74e9187713e8b82c9a23b
0905 19:52:36.285672 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.285736 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3461106}
I20250905 19:52:36.287221 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 166) took 3465 ms. Trace:
I20250905 19:52:36.287307 2379 rpcz_store.cc:276] 0905 19:52:32.821533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.824602 (+ 3069us) service_pool.cc:225] Handling call
0905 19:52:36.287144 (+3462542us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:36.287154 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.287207 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462381}
I20250905 19:52:36.287575 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 167) took 3465 ms. Trace:
I20250905 19:52:36.287642 2379 rpcz_store.cc:276] 0905 19:52:32.821649 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.824794 (+ 3145us) service_pool.cc:225] Handling call
0905 19:52:36.287509 (+3462715us) ts_tablet_manager.cc:1145] Deleting tablet 9e9994863c374354838aa0f817d0057e
0905 19:52:36.287517 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.287562 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462574}
I20250905 19:52:36.287878 2379 tablet_replica.cc:331] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.288079 2379 raft_consensus.cc:2241] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.288267 2379 raft_consensus.cc:2270] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.293697 2379 ts_tablet_manager.cc:1905] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.294929 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.300277 2930 consensus_queue.cc:1035] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.300971 3063 raft_consensus.cc:491] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.301141 3063 raft_consensus.cc:513] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.301883 3063 leader_election.cc:290] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.302706 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.302920 1051 raft_consensus.cc:2391] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:36.303694 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:36.309199 984 consensus_peers.cc:489] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.310245 1171 tablet_bootstrap.cc:492] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.311048 1171 ts_tablet_manager.cc:1397] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.149s user 0.010s sys 0.010s
I20250905 19:52:36.315062 2379 ts_tablet_manager.cc:1918] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.315189 2379 log.cc:1199] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/3de936d2309a411eabbd59f7b73ce7f3
I20250905 19:52:36.315570 2379 ts_tablet_manager.cc:1939] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.315953 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 169) took 3494 ms. Trace:
I20250905 19:52:36.316111 2379 rpcz_store.cc:276] 0905 19:52:32.821874 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.824963 (+ 3089us) service_pool.cc:225] Handling call
0905 19:52:36.287835 (+3462872us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:36.287843 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.308442 (+ 20599us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.314943 (+ 6501us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.315930 (+ 987us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3462725}
I20250905 19:52:36.316531 2379 tablet_replica.cc:331] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.316726 2379 raft_consensus.cc:2241] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.316924 2379 raft_consensus.cc:2270] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.318859 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 89) took 6865 ms. Trace:
I20250905 19:52:36.319036 2376 rpcz_store.cc:276] 0905 19:52:29.453560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.455013 (+ 1453us) service_pool.cc:225] Handling call
0905 19:52:36.181760 (+6726747us) ts_tablet_manager.cc:1145] Deleting tablet 421b28d047124222ae9cbc603ec22cee
0905 19:52:36.181772 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.199709 (+ 17937us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.264059 (+ 64350us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.318836 (+ 54777us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":12271,"tablet-delete.queue_time_us":6726577}
I20250905 19:52:36.319581 2376 tablet_replica.cc:331] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.319898 2376 raft_consensus.cc:2241] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.320166 2376 raft_consensus.cc:2270] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.320919 2376 ts_tablet_manager.cc:1905] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.321760 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 421b28d047124222ae9cbc603ec22cee (table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]) successfully deleted
I20250905 19:52:36.323447 2379 ts_tablet_manager.cc:1905] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.334915 1171 raft_consensus.cc:357] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.335327 1171 raft_consensus.cc:383] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.335443 1171 raft_consensus.cc:738] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.335888 1171 consensus_queue.cc:260] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.337356 1171 ts_tablet_manager.cc:1428] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20250905 19:52:36.337740 1171 tablet_bootstrap.cc:492] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.340003 2379 ts_tablet_manager.cc:1918] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.340128 2379 log.cc:1199] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/c14dc571b586405f8fd1eaa452998978
I20250905 19:52:36.340471 2379 ts_tablet_manager.cc:1939] T c14dc571b586405f8fd1eaa452998978 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.343320 2379 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:54786 (request call id 170) took 2334 ms. Trace:
I20250905 19:52:36.343446 2379 rpcz_store.cc:276] 0905 19:52:34.008771 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.008839 (+ 68us) service_pool.cc:225] Handling call
0905 19:52:36.316466 (+2307627us) ts_tablet_manager.cc:1145] Deleting tablet c14dc571b586405f8fd1eaa452998978
0905 19:52:36.316478 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.331549 (+ 15071us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.339859 (+ 8310us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.343291 (+ 3432us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2307394}
I20250905 19:52:36.346629 3063 raft_consensus.cc:491] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.346800 3063 raft_consensus.cc:513] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.347247 848 leader_election.cc:304] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:36.347492 2422 raft_consensus.cc:2802] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.347550 2422 raft_consensus.cc:491] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.347541 3063 leader_election.cc:290] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.347604 2422 raft_consensus.cc:3058] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.348598 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.348968 1047 raft_consensus.cc:2391] T 98e4e6d274574c0580315dcc2323c3d6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:36.349426 2376 ts_tablet_manager.cc:1918] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.349550 2376 log.cc:1199] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/24bcd21f19ba465aa550102e0fccb674
I20250905 19:52:36.349862 2376 ts_tablet_manager.cc:1939] T 24bcd21f19ba465aa550102e0fccb674 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.350332 2422 raft_consensus.cc:513] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.350999 2422 leader_election.cc:290] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.351450 1171 tablet_bootstrap.cc:654] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.352414 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 91) took 6898 ms. Trace:
I20250905 19:52:36.352505 2376 rpcz_store.cc:276] 0905 19:52:29.453794 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.455220 (+ 1426us) service_pool.cc:225] Handling call
0905 19:52:36.319513 (+6864293us) ts_tablet_manager.cc:1145] Deleting tablet 24bcd21f19ba465aa550102e0fccb674
0905 19:52:36.319530 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.324894 (+ 5364us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.349312 (+ 24418us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.352393 (+ 3081us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6864128}
I20250905 19:52:36.354188 2379 tablet_replica.cc:331] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.354465 2379 raft_consensus.cc:2241] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.355213 3063 raft_consensus.cc:491] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.355360 3063 raft_consensus.cc:513] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.356084 3063 leader_election.cc:290] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.357331 1047 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.357702 1047 raft_consensus.cc:2391] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:36.358461 2379 raft_consensus.cc:2270] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.359396 2379 ts_tablet_manager.cc:1905] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.354224 2422 raft_consensus.cc:491] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.358464 2738 raft_consensus.cc:491] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.358592 2413 raft_consensus.cc:491] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.368232 2376 tablet_replica.cc:331] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.368301 2413 raft_consensus.cc:513] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.368459 2376 raft_consensus.cc:2241] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.368707 2376 raft_consensus.cc:2270] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.368866 2413 leader_election.cc:290] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.359514 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a311f9b943d143a9a52d7771ce7186d7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.359808 912 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3fe2b75a3e59437c885abc18614f1da2" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.360176 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "db098a883a61474fb06cb166041ee3e4" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.360448 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cd5312852fe345d98039aeea241b7fe8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.360695 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d011d1ec8f634386872e7ab257de3187" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.361317 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f953a7490434bc7afa5510c33e79401" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.361605 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98e4e6d274574c0580315dcc2323c3d6" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.361685 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3609de83732d4ad8857372acfbb5c32a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.369762 2376 ts_tablet_manager.cc:1905] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.368240 2738 raft_consensus.cc:513] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.370222 2738 leader_election.cc:290] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.370610 2422 raft_consensus.cc:513] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.371109 2422 leader_election.cc:290] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.371522 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 24bcd21f19ba465aa550102e0fccb674 (table default.Table_65d1d80076a24f24a1c5284f9e90ec10 [id=c7b558dcabda400d9ba6e1850e0a61eb]) successfully deleted
I20250905 19:52:36.362617 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad0760dfee934aacbd4ab1fb81e116c7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.372238 2998 raft_consensus.cc:491] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:36.372439 2998 raft_consensus.cc:513] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.373411 2998 leader_election.cc:290] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:36.373996 986 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3de936d2309a411eabbd59f7b73ce7f3
W20250905 19:52:36.363462 848 leader_election.cc:343] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.375989 2379 ts_tablet_manager.cc:1918] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.376134 2379 log.cc:1199] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/dc95a83ca872495894ef439cdda4e364
I20250905 19:52:36.376556 2379 ts_tablet_manager.cc:1939] T dc95a83ca872495894ef439cdda4e364 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:36.377751 984 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 3de936d2309a411eabbd59f7b73ce7f3
I20250905 19:52:36.377871 984 leader_election.cc:304] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.379742 3075 sys_catalog.cc:455] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "e38333af29db4bf6af41ee5a25d6701e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:36.379880 2379 tablet_replica.cc:331] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.379989 3075 sys_catalog.cc:458] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:36.380489 2379 raft_consensus.cc:2241] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.380903 2379 raft_consensus.cc:2270] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.381734 2379 ts_tablet_manager.cc:1905] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:36.385443 1472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.61:41317, user_credentials={real_user=slave}} blocked reactor thread for 151893us
I20250905 19:52:36.386409 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 2350bd08f977424ba94ee8fa01f66efa (table default.Table_9067a9bea2cc48ecab23b9fd1749474c [id=6608cae33460403c8748316259e407c5]) successfully deleted
W20250905 19:52:36.386647 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 01c20eea1b024d7f93f18f0c089efc42 because the tablet was not found. No further retry: Not found: Tablet not found: 01c20eea1b024d7f93f18f0c089efc42
W20250905 19:52:36.387092 847 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
W20250905 19:52:36.387336 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 54628965b82e470d811eff95ffc56c5f because the tablet was not found. No further retry: Not found: Tablet not found: 54628965b82e470d811eff95ffc56c5f
W20250905 19:52:36.387518 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 78076f6534814939ae8c2db9a4bd1879 because the tablet was not found. No further retry: Not found: Tablet not found: 78076f6534814939ae8c2db9a4bd1879
W20250905 19:52:36.387728 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ab4f69c45295422f85b310aa55220e9b because the tablet was not found. No further retry: Not found: Tablet not found: ab4f69c45295422f85b310aa55220e9b
W20250905 19:52:36.387872 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 421b28d047124222ae9cbc603ec22cee already in progress: creating tablet
W20250905 19:52:36.388053 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2683a36998674f739b41d2a57339954c because the tablet was not found. No further retry: Not found: Tablet not found: 2683a36998674f739b41d2a57339954c
W20250905 19:52:36.388188 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet f28e403dc2f6471ebea1805fd843cfb1 because the tablet was not found. No further retry: Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1
W20250905 19:52:36.388396 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 05ae83eea5f74e9187713e8b82c9a23b because the tablet was not found. No further retry: Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b
W20250905 19:52:36.388705 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 5b0d4be788f841c2860f5dd65275b623 because the tablet was not found. No further retry: Not found: Tablet not found: 5b0d4be788f841c2860f5dd65275b623
W20250905 19:52:36.388883 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9e9994863c374354838aa0f817d0057e because the tablet was not found. No further retry: Not found: Tablet not found: 9e9994863c374354838aa0f817d0057e
I20250905 19:52:36.389005 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 3de936d2309a411eabbd59f7b73ce7f3 (table default.Table_88ff02c5dce2458784f09b64cee688f1 [id=378b42fcb4e741e4b4755deb4b0ef51a]) successfully deleted
I20250905 19:52:36.389142 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet c14dc571b586405f8fd1eaa452998978 (table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]) successfully deleted
W20250905 19:52:36.389302 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 because the tablet was not found. No further retry: Not found: Tablet not found: 37894e39e6f2417fbfcd6d43f1ea2b37
W20250905 19:52:36.389477 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e because the tablet was not found. No further retry: Not found: Tablet not found: cbfd03ad0bd34b6ca86e7b07a62b2a8e
W20250905 19:52:36.389657 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet f48f9f789b6e46ec871a0fd007770dfc because the tablet was not found. No further retry: Not found: Tablet not found: f48f9f789b6e46ec871a0fd007770dfc
I20250905 19:52:36.396373 1170 tablet_bootstrap.cc:492] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.396524 1170 ts_tablet_manager.cc:1397] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.251s user 0.018s sys 0.027s
I20250905 19:52:36.396420 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9d9815864aa7bf5ca9a5c4485390" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:36.396695 768 raft_consensus.cc:3058] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.397233 1170 raft_consensus.cc:357] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.397392 1170 raft_consensus.cc:383] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.397447 1170 raft_consensus.cc:738] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.397635 1170 consensus_queue.cc:260] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.398069 1170 ts_tablet_manager.cc:1428] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:36.400348 768 raft_consensus.cc:2466] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:36.402498 1169 tablet_bootstrap.cc:654] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.403509 2998 raft_consensus.cc:2747] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:36.405382 768 raft_consensus.cc:3058] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.406271 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.406543 770 raft_consensus.cc:3058] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.407224 1170 tablet_bootstrap.cc:492] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.408975 1170 tablet_bootstrap.cc:654] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.409514 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.410099 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95b32fff7d4745c79185a5d894ef3342" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.410302 766 raft_consensus.cc:2466] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:36.421322 2376 ts_tablet_manager.cc:1918] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.421474 2376 log.cc:1199] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/cbfd03ad0bd34b6ca86e7b07a62b2a8e
I20250905 19:52:36.425160 2903 raft_consensus.cc:491] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.425365 2903 raft_consensus.cc:513] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.427479 848 leader_election.cc:304] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:36.427771 2422 raft_consensus.cc:2802] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:36.428261 2422 raft_consensus.cc:695] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:36.429220 2903 leader_election.cc:290] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.430579 1171 tablet_bootstrap.cc:492] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.430778 1171 ts_tablet_manager.cc:1397] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.093s user 0.011s sys 0.000s
W20250905 19:52:36.431325 712 leader_election.cc:343] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:36.428572 2422 consensus_queue.cc:237] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.431483 712 leader_election.cc:304] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:36.431651 712 raft_consensus.cc:2684] T 06850faa25de4b3d83f6bf98ccd370f0 P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:36.434321 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.435201 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c77dd068ea5c495ca8aa0e7a6823d792" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.435667 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10b4e82c4bc429b9a42db39c03a5267" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.436139 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.437439 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9d9815864aa7bf5ca9a5c4485390" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.437587 1051 raft_consensus.cc:2466] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:36.437624 1171 raft_consensus.cc:357] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.437788 1171 raft_consensus.cc:383] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.437840 1171 raft_consensus.cc:738] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.438066 1171 consensus_queue.cc:260] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.438191 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.441608 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6ac13b19fb6427a9ef35a5160033a54" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.442267 1171 ts_tablet_manager.cc:1428] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250905 19:52:36.442642 1171 tablet_bootstrap.cc:492] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.444748 2376 ts_tablet_manager.cc:1939] T cbfd03ad0bd34b6ca86e7b07a62b2a8e P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:36.445371 984 consensus_peers.cc:489] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.445367 3063 raft_consensus.cc:491] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.445580 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 92) took 6991 ms. Trace:
I20250905 19:52:36.445538 3063 raft_consensus.cc:513] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.445713 2376 rpcz_store.cc:276] 0905 19:52:29.453945 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.455405 (+ 1460us) service_pool.cc:225] Handling call
0905 19:52:36.368156 (+6912751us) ts_tablet_manager.cc:1145] Deleting tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
0905 19:52:36.368171 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.381971 (+ 13800us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.421132 (+ 39161us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.445529 (+ 24397us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6912600}
I20250905 19:52:36.446063 2376 tablet_replica.cc:331] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.446159 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e (table default.Table_c7c4cd0bb1b14ead83f2942518f4c30f [id=9049c258ecc54305be2e9641520fb9ba]) successfully deleted
I20250905 19:52:36.446285 2376 raft_consensus.cc:2241] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.446332 3063 leader_election.cc:290] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.446442 2376 raft_consensus.cc:2270] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.446985 2376 ts_tablet_manager.cc:1905] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.447501 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.447767 1044 raft_consensus.cc:2391] T 3b53259c71d5443ebb12827ee78038fe P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:36.448624 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5fdc9d9815864aa7bf5ca9a5c4485390" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:36.448794 1051 raft_consensus.cc:3058] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.451383 1051 raft_consensus.cc:2466] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:36.460389 960 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:36.460397 2379 ts_tablet_manager.cc:1918] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.460517 2379 log.cc:1199] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:36.462417 2379 ts_tablet_manager.cc:1939] T 8d38452d3a0641ffa4af5d935f8ad9f0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.463740 1543 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 54) took 8188 ms. Trace:
I20250905 19:52:36.463907 1543 rpcz_store.cc:276] 0905 19:52:28.275623 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.010255 (+7734632us) service_pool.cc:225] Handling call
0905 19:52:36.011765 (+ 1510us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:36.012273 (+ 508us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:36.463440 (+451167us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:36.463564 (+ 124us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:36.463690 (+ 126us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.044542 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.044746 (+ 204us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.044752 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:36.044754 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:36.044920 (+ 166us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.044936 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.044942 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:36.045034 (+ 92us) tablet.cc:699] Row locks acquired
0905 19:52:36.045039 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:36.045086 (+ 47us) write_op.cc:270] Start()
0905 19:52:36.045132 (+ 46us) write_op.cc:276] Timestamp: P: 1757101956045066 usec, L: 0
0905 19:52:36.045138 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.045382 (+ 244us) log.cc:844] Serialized 433 byte log entry
0905 19:52:36.390062 (+344680us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.462550 (+ 72488us) write_op.cc:301] APPLY: starting
0905 19:52:36.462578 (+ 28us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.462636 (+ 58us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.462697 (+ 61us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.462871 (+ 174us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.462909 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:36.462917 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:36.463023 (+ 106us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.463314 (+ 291us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.463346 (+ 32us) write_op.cc:454] Released schema lock
0905 19:52:36.463357 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":72343,"apply.run_cpu_time_us":691,"apply.run_wall_time_us":954,"num_ops":2,"prepare.queue_time_us":31674,"prepare.run_cpu_time_us":988,"prepare.run_wall_time_us":3676,"raft.queue_time_us":218,"raft.run_cpu_time_us":279,"raft.run_wall_time_us":278,"replication_time_us":344888,"thread_start_us":108615,"threads_started":2}]]}
I20250905 19:52:36.465014 1543 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_e065514fff894d6c837edd542a939abd" } modify_external_catalogs: true
I20250905 19:52:36.465211 1543 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_e065514fff894d6c837edd542a939abd" } modify_external_catalogs: true
W20250905 19:52:36.466562 848 leader_election.cc:343] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.467113 2838 consensus_queue.cc:1035] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:36.468031 712 leader_election.cc:343] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:36.468163 712 leader_election.cc:304] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.468686 3063 raft_consensus.cc:2747] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:36.473637 712 leader_election.cc:343] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3de936d2309a411eabbd59f7b73ce7f3
I20250905 19:52:36.473766 712 leader_election.cc:304] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:36.473922 712 raft_consensus.cc:2684] T 3de936d2309a411eabbd59f7b73ce7f3 P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:36.476130 1171 tablet_bootstrap.cc:654] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.481329 984 consensus_peers.cc:489] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:36.483034 984 consensus_peers.cc:489] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.483991 1170 tablet_bootstrap.cc:492] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.484175 1170 ts_tablet_manager.cc:1397] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.003s
I20250905 19:52:36.485113 1170 raft_consensus.cc:357] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.485311 1171 tablet_bootstrap.cc:492] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.485352 1170 raft_consensus.cc:383] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.485430 1170 raft_consensus.cc:738] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.485486 1171 ts_tablet_manager.cc:1397] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.043s user 0.003s sys 0.000s
I20250905 19:52:36.485695 1170 consensus_queue.cc:260] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.486311 1171 raft_consensus.cc:357] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.486414 848 leader_election.cc:343] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.486845 848 leader_election.cc:304] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:36.487655 2903 raft_consensus.cc:2802] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.487768 2903 raft_consensus.cc:491] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.487857 2903 raft_consensus.cc:3058] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.488255 2379 tablet_replica.cc:331] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.489949 1170 ts_tablet_manager.cc:1428] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:36.490312 1170 tablet_bootstrap.cc:492] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.490478 2903 raft_consensus.cc:513] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.491048 2903 leader_election.cc:290] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.492153 2379 raft_consensus.cc:2241] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.492324 2379 raft_consensus.cc:2270] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.492885 2379 ts_tablet_manager.cc:1905] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.494697 1169 tablet_bootstrap.cc:492] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:36.494940 1169 ts_tablet_manager.cc:1397] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.281s user 0.000s sys 0.004s
I20250905 19:52:36.495793 1169 raft_consensus.cc:357] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.496037 1169 raft_consensus.cc:383] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.496131 1169 raft_consensus.cc:738] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.496412 1169 consensus_queue.cc:260] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.497690 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9db56577a26e407b97fc33158c2c466c because the tablet was not found. No further retry: Not found: Tablet not found: 9db56577a26e407b97fc33158c2c466c
W20250905 19:52:36.498096 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3602094b96c14e66b997e8035fc53c40 because the tablet was not found. No further retry: Not found: Tablet not found: 3602094b96c14e66b997e8035fc53c40
W20250905 19:52:36.498504 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because the tablet was not found. No further retry: Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
W20250905 19:52:36.498837 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd1280fefc3e43db9928567a734644f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1280fefc3e43db9928567a734644f1 already in progress: creating tablet
W20250905 19:52:36.499076 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ac5c88cf8d15433a940e937f0a225582 because the tablet was not found. No further retry: Not found: Tablet not found: ac5c88cf8d15433a940e937f0a225582
W20250905 19:52:36.499326 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 03a12cf719f34096bb82f5846bbe3b2d because the tablet was not found. No further retry: Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
W20250905 19:52:36.499569 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a8b6823580cb4040b49ed0a49b748965 because the tablet was not found. No further retry: Not found: Tablet not found: a8b6823580cb4040b49ed0a49b748965
W20250905 19:52:36.499789 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 00ac56c449ba4b7b8dbd5e87120799d9 because the tablet was not found. No further retry: Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9
W20250905 19:52:36.500123 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 42f3c1efb94e446fb0128b6b412821e2 because the tablet was not found. No further retry: Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2
W20250905 19:52:36.501402 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9630f1feb6d74472b5f6f4b54151311c because the tablet was not found. No further retry: Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:36.502753 2376 ts_tablet_manager.cc:1918] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.502902 2376 log.cc:1199] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/a1b4989f17454aa58c05727b797c2009
I20250905 19:52:36.503239 2376 ts_tablet_manager.cc:1939] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.503585 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 94) took 7049 ms. Trace:
I20250905 19:52:36.503682 2376 rpcz_store.cc:276] 0905 19:52:29.454159 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.455574 (+ 1415us) service_pool.cc:225] Handling call
0905 19:52:36.445997 (+6990423us) ts_tablet_manager.cc:1145] Deleting tablet a1b4989f17454aa58c05727b797c2009
0905 19:52:36.446009 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.479181 (+ 33172us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.502625 (+ 23444us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.503526 (+ 901us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":6990297}
I20250905 19:52:36.486521 1171 raft_consensus.cc:383] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.504148 1171 raft_consensus.cc:738] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.504428 1171 consensus_queue.cc:260] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.505215 1171 ts_tablet_manager.cc:1428] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250905 19:52:36.505524 1171 tablet_bootstrap.cc:492] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.507306 1171 tablet_bootstrap.cc:654] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.508781 768 raft_consensus.cc:3058] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.508881 775 raft_consensus.cc:3058] T 1d095804b51b4205a23ac3d3106312f2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.511047 3063 raft_consensus.cc:491] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.511175 3063 raft_consensus.cc:513] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.511751 3063 leader_election.cc:290] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.512123 776 raft_consensus.cc:3058] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.512984 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.513098 2379 ts_tablet_manager.cc:1918] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.513196 2379 log.cc:1199] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/3609de83732d4ad8857372acfbb5c32a
I20250905 19:52:36.513504 2379 ts_tablet_manager.cc:1939] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.513377 1044 raft_consensus.cc:2391] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:36.514039 2379 tablet_replica.cc:331] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:36.514225 2379 raft_consensus.cc:2241] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.514370 2379 raft_consensus.cc:2270] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.514868 2379 ts_tablet_manager.cc:1905] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.516830 1170 tablet_bootstrap.cc:654] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.518450 2652 ts_tablet_manager.cc:1918] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250905 19:52:36.518613 2652 log.cc:1199] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/2683a36998674f739b41d2a57339954c
I20250905 19:52:36.519423 2652 ts_tablet_manager.cc:1939] T 2683a36998674f739b41d2a57339954c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:36.519783 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 83) took 4897 ms. Trace:
I20250905 19:52:36.520747 849 leader_election.cc:304] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.521061 2413 raft_consensus.cc:2747] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.523545 849 leader_election.cc:304] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.524907 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95b32fff7d4745c79185a5d894ef3342" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:36.525065 767 raft_consensus.cc:3058] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.528133 767 raft_consensus.cc:2466] T 95b32fff7d4745c79185a5d894ef3342 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:36.533028 2413 raft_consensus.cc:2802] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.533159 2413 raft_consensus.cc:491] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.533274 2413 raft_consensus.cc:3058] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.534459 1170 tablet_bootstrap.cc:492] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.534636 1170 ts_tablet_manager.cc:1397] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.002s
I20250905 19:52:36.535429 2376 tablet_replica.cc:331] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.535658 2376 raft_consensus.cc:2241] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.535591 1170 raft_consensus.cc:357] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.535796 1170 raft_consensus.cc:383] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.535856 1170 raft_consensus.cc:738] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.536111 1170 consensus_queue.cc:260] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.536245 2376 raft_consensus.cc:2270] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.536716 1170 ts_tablet_manager.cc:1428] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:36.536823 2376 ts_tablet_manager.cc:1905] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.537048 1170 tablet_bootstrap.cc:492] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.536866 2413 raft_consensus.cc:513] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.537730 2413 leader_election.cc:290] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.538113 2413 raft_consensus.cc:491] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.538306 2413 raft_consensus.cc:513] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.538853 2413 leader_election.cc:290] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.541728 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:36.542526 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a860ab3f5ce54bd2b75bb71730a3c6e6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.504163 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet a1b4989f17454aa58c05727b797c2009 (table default.Table_ee3eaf9eb84d435187c73a394308861b [id=8225e2fcccbf436db5bfa9e05ba88f83]) successfully deleted
W20250905 19:52:36.548516 1484 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 44381.9us
I20250905 19:52:36.548825 1171 tablet_bootstrap.cc:492] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.548971 1171 ts_tablet_manager.cc:1397] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
W20250905 19:52:36.548969 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 47491.8us
W20250905 19:52:36.549484 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2f581403ec924fb0948400df41ed8aab because the tablet was not found. No further retry: Not found: Tablet not found: 2f581403ec924fb0948400df41ed8aab
I20250905 19:52:36.549748 1171 raft_consensus.cc:357] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.549911 1171 raft_consensus.cc:383] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.550096 1171 raft_consensus.cc:738] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.550287 1171 consensus_queue.cc:260] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.550789 1171 ts_tablet_manager.cc:1428] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:36.551062 1171 tablet_bootstrap.cc:492] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.551832 1170 tablet_bootstrap.cc:654] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.551707 2652 rpcz_store.cc:276] 0905 19:52:31.622525 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.622824 (+ 299us) service_pool.cc:225] Handling call
0905 19:52:35.935640 (+4312816us) ts_tablet_manager.cc:1145] Deleting tablet 2683a36998674f739b41d2a57339954c
0905 19:52:35.935651 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.413137 (+477486us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.518340 (+105203us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.519769 (+ 1429us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4309363}
I20250905 19:52:36.552435 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 2683a36998674f739b41d2a57339954c (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]) successfully deleted
I20250905 19:52:36.552781 2652 tablet_replica.cc:331] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:36.553013 2652 raft_consensus.cc:2241] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.553408 2652 raft_consensus.cc:2270] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.554083 2652 ts_tablet_manager.cc:1905] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.521540 1169 ts_tablet_manager.cc:1428] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.026s user 0.001s sys 0.001s
I20250905 19:52:36.532446 2930 raft_consensus.cc:491] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 835daf30e4324782848290be181ae78f)
I20250905 19:52:36.555794 2930 raft_consensus.cc:513] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.556648 2930 leader_election.cc:290] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:36.557211 984 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:36.557308 1171 tablet_bootstrap.cc:654] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.539049 2998 raft_consensus.cc:491] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.551375 2496 consensus_queue.cc:1035] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.558176 2998 raft_consensus.cc:513] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.559165 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:36.559228 2998 leader_election.cc:290] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.559368 767 raft_consensus.cc:2466] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:36.559736 984 leader_election.cc:304] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:36.560565 2998 raft_consensus.cc:2802] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.560691 2998 raft_consensus.cc:491] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.560825 2998 raft_consensus.cc:3058] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.561808 1170 tablet_bootstrap.cc:492] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.561949 1170 ts_tablet_manager.cc:1397] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.025s user 0.003s sys 0.000s
I20250905 19:52:36.562610 1170 raft_consensus.cc:357] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.562801 1170 raft_consensus.cc:383] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.562871 1170 raft_consensus.cc:738] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.563081 1170 consensus_queue.cc:260] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.563480 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.563620 1170 ts_tablet_manager.cc:1428] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:36.563639 907 raft_consensus.cc:2466] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:36.565351 2998 raft_consensus.cc:513] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.566510 986 leader_election.cc:343] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:36.566615 986 leader_election.cc:304] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.566936 2496 raft_consensus.cc:2747] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:36.567544 1169 tablet_bootstrap.cc:492] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:36.573309 986 consensus_peers.cc:489] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3c824f5591044c4198a51a25661f71e5. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.574522 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:36.574829 906 raft_consensus.cc:3058] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.574934 2379 ts_tablet_manager.cc:1918] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.575057 2379 log.cc:1199] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/a2ff206366ad4344b4df5fc6a5584f8a
I20250905 19:52:36.575376 2379 ts_tablet_manager.cc:1939] T a2ff206366ad4344b4df5fc6a5584f8a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:36.575610 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:36.575801 770 raft_consensus.cc:3058] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.580998 906 raft_consensus.cc:2466] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:36.581362 2998 leader_election.cc:290] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.581686 770 raft_consensus.cc:2466] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:36.582193 984 leader_election.cc:304] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:36.582631 2998 raft_consensus.cc:2802] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
W20250905 19:52:36.581974 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 49097.4us
I20250905 19:52:36.585228 1171 tablet_bootstrap.cc:492] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.585422 1171 ts_tablet_manager.cc:1397] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.000s
I20250905 19:52:36.585644 2998 raft_consensus.cc:695] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:36.586190 1171 raft_consensus.cc:357] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.586357 1171 raft_consensus.cc:383] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.586409 1171 raft_consensus.cc:738] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.586654 849 leader_election.cc:304] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.587037 2998 consensus_queue.cc:237] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.587445 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a5f076d9917f40d899d7422ad974fe0f because the tablet was not found. No further retry: Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f
W20250905 19:52:36.587730 849 raft_consensus.cc:2684] T fd1280fefc3e43db9928567a734644f1 P 835daf30e4324782848290be181ae78f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:36.586625 1171 consensus_queue.cc:260] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.587922 848 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.104.61:41317, user_credentials={real_user=slave}} blocked reactor thread for 44904.2us
W20250905 19:52:36.590363 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 00d8144bb4f14358af539b7aae326918 because the tablet was not found. No further retry: Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918
W20250905 19:52:36.591166 848 consensus_peers.cc:489] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c14dc571b586405f8fd1eaa452998978. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.563926 1170 tablet_bootstrap.cc:492] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.600719 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]) successfully deleted
I20250905 19:52:36.601511 1170 tablet_bootstrap.cc:654] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.602273 1171 ts_tablet_manager.cc:1428] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.017s user 0.002s sys 0.000s
I20250905 19:52:36.602371 822 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38687
I20250905 19:52:36.602600 1171 tablet_bootstrap.cc:492] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.589977 1169 tablet_bootstrap.cc:654] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.604074 1171 tablet_bootstrap.cc:654] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.602576 848 consensus_peers.cc:489] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:36.604844 848 leader_election.cc:343] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Not found: Tablet not found: 9db56577a26e407b97fc33158c2c466c
I20250905 19:52:36.605612 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8d38452d3a0641ffa4af5d935f8ad9f0 (table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]) successfully deleted
I20250905 19:52:36.607378 2738 consensus_queue.cc:1035] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:36.610556 984 consensus_peers.cc:489] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.611085 2496 consensus_queue.cc:1035] T 2568b9f6f37b48efbe92cef4c19d5dfb P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.621071 1171 tablet_bootstrap.cc:492] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.621234 1171 ts_tablet_manager.cc:1397] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
W20250905 19:52:36.621795 848 consensus_peers.cc:489] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc95a83ca872495894ef439cdda4e364. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.624454 2789 consensus_queue.cc:1035] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:36.608716 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 53ac46ce05694bc2b53e195638be9e23 because the tablet was not found. No further retry: Not found: Tablet not found: 53ac46ce05694bc2b53e195638be9e23
W20250905 19:52:36.626432 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 60a41dbd8782460cb0d3c5f8bd59354f because the tablet was not found. No further retry: Not found: Tablet not found: 60a41dbd8782460cb0d3c5f8bd59354f
I20250905 19:52:36.627044 2376 ts_tablet_manager.cc:1918] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.627149 2376 log.cc:1199] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/f48f9f789b6e46ec871a0fd007770dfc
I20250905 19:52:36.627501 2376 ts_tablet_manager.cc:1939] T f48f9f789b6e46ec871a0fd007770dfc P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.627828 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 90) took 7174 ms. Trace:
I20250905 19:52:36.628440 2738 consensus_queue.cc:1035] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.628749 1171 raft_consensus.cc:357] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.628886 1171 raft_consensus.cc:383] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.628932 1171 raft_consensus.cc:738] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.629091 1171 consensus_queue.cc:260] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.629420 848 leader_election.cc:304] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:36.629581 1171 ts_tablet_manager.cc:1428] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.000s sys 0.001s
I20250905 19:52:36.629745 2789 raft_consensus.cc:2802] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:36.629855 1171 tablet_bootstrap.cc:492] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.629860 2789 raft_consensus.cc:695] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:36.630185 2789 consensus_queue.cc:237] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.631274 1171 tablet_bootstrap.cc:654] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.636062 2405 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" instance_seqno: 1757101932934030) as {username='slave'} at 127.0.104.1:45391; Asking this server to re-register.
I20250905 19:52:36.636919 1170 tablet_bootstrap.cc:492] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.637068 1170 ts_tablet_manager.cc:1397] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.073s user 0.000s sys 0.005s
I20250905 19:52:36.637908 1170 raft_consensus.cc:357] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.638103 1170 raft_consensus.cc:383] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.638161 1170 raft_consensus.cc:738] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.638389 1170 consensus_queue.cc:260] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.638928 1170 ts_tablet_manager.cc:1428] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:36.639195 1170 tablet_bootstrap.cc:492] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.640760 2738 consensus_queue.cc:1035] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.640878 1170 tablet_bootstrap.cc:654] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.641279 849 leader_election.cc:343] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.641391 849 leader_election.cc:304] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.649690 822 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:36.652925 1169 tablet_bootstrap.cc:492] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:36.653074 1169 ts_tablet_manager.cc:1397] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.086s user 0.003s sys 0.000s
I20250905 19:52:36.653889 1169 raft_consensus.cc:357] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.654083 1169 raft_consensus.cc:383] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.654134 1169 raft_consensus.cc:738] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.654383 1169 consensus_queue.cc:260] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.655011 1169 ts_tablet_manager.cc:1428] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:36.655263 1169 tablet_bootstrap.cc:492] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:36.656714 1170 tablet_bootstrap.cc:492] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.656859 1170 ts_tablet_manager.cc:1397] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.000s
I20250905 19:52:36.657450 1170 raft_consensus.cc:357] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.657627 1170 raft_consensus.cc:383] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.657684 1170 raft_consensus.cc:738] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.657862 1170 consensus_queue.cc:260] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.658327 1170 ts_tablet_manager.cc:1428] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250905 19:52:36.658566 1170 tablet_bootstrap.cc:492] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.660055 1170 tablet_bootstrap.cc:654] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.660732 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f because the tablet was not found. No further retry: Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f
W20250905 19:52:36.660964 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9193487b4ab74dafa76a386ddcaf674b because the tablet was not found. No further retry: Not found: Tablet not found: 9193487b4ab74dafa76a386ddcaf674b
W20250905 19:52:36.661149 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 723168b075874dfcb3eb50c5134c499e because the tablet was not found. No further retry: Not found: Tablet not found: 723168b075874dfcb3eb50c5134c499e
W20250905 19:52:36.661317 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a3805127dd78489e8545da7414ead904 because the tablet was not found. No further retry: Not found: Tablet not found: a3805127dd78489e8545da7414ead904
W20250905 19:52:36.661499 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 2d84653c9e7a4857a7580fa85f8e4732 because the tablet was not found. No further retry: Not found: Tablet not found: 2d84653c9e7a4857a7580fa85f8e4732
W20250905 19:52:36.661654 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6f3473785fff45d692bd50b1015ffe39 because the tablet was not found. No further retry: Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39
W20250905 19:52:36.661825 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet ba161282480e4736889559a2e37b195f because the tablet was not found. No further retry: Not found: Tablet not found: ba161282480e4736889559a2e37b195f
W20250905 19:52:36.661969 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 24bcd21f19ba465aa550102e0fccb674 because the tablet was not found. No further retry: Not found: Tablet not found: 24bcd21f19ba465aa550102e0fccb674
W20250905 19:52:36.662130 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 3c824f5591044c4198a51a25661f71e5 already in progress: creating tablet
W20250905 19:52:36.662541 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because the tablet was not found. No further retry: Not found: Tablet not found: 1f1a8e7059b348cba34069ef49bf1aad
W20250905 19:52:36.662781 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because the tablet was not found. No further retry: Not found: Tablet not found: 578a6b9224cc4d8585d6c4b241762e1b
W20250905 19:52:36.662971 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 302053e1611b4e22a9a2e7b60ae68f27 because the tablet was not found. No further retry: Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:36.663141 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 3609de83732d4ad8857372acfbb5c32a (table default.Table_f6e643730c10465cb8585cb48c1c5e04 [id=fa983711b6574ebeb9dfb2cca302bdc1]) successfully deleted
I20250905 19:52:36.663318 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet a2ff206366ad4344b4df5fc6a5584f8a (table default.Table_fe03bfebc206445e842984b466de6d91 [id=4aaef69a757d4b179b91be552c4eacc0]) successfully deleted
W20250905 19:52:36.663513 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd1280fefc3e43db9928567a734644f1 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet fd1280fefc3e43db9928567a734644f1 already in progress: creating tablet
W20250905 19:52:36.663686 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a1b4989f17454aa58c05727b797c2009 because the tablet was not found. No further retry: Not found: Tablet not found: a1b4989f17454aa58c05727b797c2009
W20250905 19:52:36.663856 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet a8a32f742eaf473e9f6f9bf214f53679 because the tablet was not found. No further retry: Not found: Tablet not found: a8a32f742eaf473e9f6f9bf214f53679
W20250905 19:52:36.664062 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 14d5eb8bec074d2c8766fe7b159031e5 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 14d5eb8bec074d2c8766fe7b159031e5 already in progress: creating tablet
I20250905 19:52:36.627903 2376 rpcz_store.cc:276] 0905 19:52:29.453683 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.455723 (+ 2040us) service_pool.cc:225] Handling call
0905 19:52:36.535324 (+7079601us) ts_tablet_manager.cc:1145] Deleting tablet f48f9f789b6e46ec871a0fd007770dfc
0905 19:52:36.535342 (+ 18us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.591616 (+ 56274us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.626930 (+ 35314us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.627811 (+ 881us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7048103}
I20250905 19:52:36.664409 2376 tablet_replica.cc:331] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.664644 2376 raft_consensus.cc:2241] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.666339 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:36.667071 2405 ts_manager.cc:194] Registered new tserver with Master: f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531)
I20250905 19:52:36.668908 1171 tablet_bootstrap.cc:492] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.669061 1171 ts_tablet_manager.cc:1397] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.039s user 0.001s sys 0.002s
I20250905 19:52:36.682773 2413 raft_consensus.cc:2747] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.683869 2422 consensus_queue.cc:1035] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.685032 1171 raft_consensus.cc:357] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.685276 1171 raft_consensus.cc:383] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.685376 1171 raft_consensus.cc:738] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.685635 1171 consensus_queue.cc:260] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.686219 1171 ts_tablet_manager.cc:1428] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.017s user 0.001s sys 0.001s
I20250905 19:52:36.686560 1171 tablet_bootstrap.cc:492] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.688596 1171 tablet_bootstrap.cc:654] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.689695 2376 raft_consensus.cc:2270] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.690357 2376 ts_tablet_manager.cc:1905] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:36.694927 848 leader_election.cc:343] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:36.695113 848 leader_election.cc:343] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.696506 3063 raft_consensus.cc:491] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.696664 3063 raft_consensus.cc:513] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.697396 3063 leader_election.cc:290] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.698176 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.699584 2998 consensus_queue.cc:1035] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:36.715235 984 consensus_peers.cc:489] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.716225 1171 tablet_bootstrap.cc:492] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.716363 1171 ts_tablet_manager.cc:1397] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250905 19:52:36.717154 770 raft_consensus.cc:3058] T 97d949c3c24e40fe8fefd3a5758d57ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.717165 1171 raft_consensus.cc:357] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.717455 1171 raft_consensus.cc:383] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.717556 1171 raft_consensus.cc:738] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.717995 1171 consensus_queue.cc:260] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.718568 1171 ts_tablet_manager.cc:1428] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:36.718851 1171 tablet_bootstrap.cc:492] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.720592 1171 tablet_bootstrap.cc:654] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.721642 771 raft_consensus.cc:3058] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.724264 767 raft_consensus.cc:3058] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.725188 2376 ts_tablet_manager.cc:1918] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.725299 2376 log.cc:1199] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6221d7d058944bdb9aa730791cf2417b
I20250905 19:52:36.725675 2376 ts_tablet_manager.cc:1939] T 6221d7d058944bdb9aa730791cf2417b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.725965 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 99) took 7117 ms. Trace:
I20250905 19:52:36.726037 2376 rpcz_store.cc:276] 0905 19:52:29.608579 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.608652 (+ 73us) service_pool.cc:225] Handling call
0905 19:52:36.664349 (+7055697us) ts_tablet_manager.cc:1145] Deleting tablet 6221d7d058944bdb9aa730791cf2417b
0905 19:52:36.664364 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.698545 (+ 34181us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.725067 (+ 26522us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.725949 (+ 882us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":24738,"tablet-delete.queue_time_us":7016038}
I20250905 19:52:36.726511 2376 tablet_replica.cc:331] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.726712 2376 raft_consensus.cc:2241] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.726992 2376 raft_consensus.cc:2270] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.727514 2376 ts_tablet_manager.cc:1905] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:36.731782 849 leader_election.cc:343] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.732687 849 leader_election.cc:304] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.734978 2422 raft_consensus.cc:2747] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:36.737254 2931 consensus_meta.cc:287] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent flushing consensus metadata: real 0.614s user 0.024s sys 0.000s
I20250905 19:52:36.737636 2422 consensus_queue.cc:1035] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.739280 3101 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:36.739385 3101 catalog_manager.cc:1554] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: loading cluster ID for follower catalog manager: success
I20250905 19:52:36.740509 2422 consensus_queue.cc:1035] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:36.741562 848 consensus_peers.cc:489] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6bb79e3126074736a23b128a8e14fab6. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.748634 2789 consensus_queue.cc:1035] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.748916 1171 tablet_bootstrap.cc:492] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.749048 1171 ts_tablet_manager.cc:1397] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.000s
I20250905 19:52:36.749823 1171 raft_consensus.cc:357] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.750002 1171 raft_consensus.cc:383] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.750058 1171 raft_consensus.cc:738] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.750258 1171 consensus_queue.cc:260] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.747035 2931 raft_consensus.cc:513] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.751623 849 leader_election.cc:343] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.751782 849 leader_election.cc:304] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.753070 2931 leader_election.cc:290] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.753010 776 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6fe4620dda4b1296416e01f8ffee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:36.753597 984 leader_election.cc:343] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.753834 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ae6fe4620dda4b1296416e01f8ffee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:36.754040 907 raft_consensus.cc:3058] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.757191 907 raft_consensus.cc:2466] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:36.757882 986 leader_election.cc:304] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.761334 2930 raft_consensus.cc:2802] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:36.762408 2930 raft_consensus.cc:695] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:36.762734 2930 consensus_queue.cc:237] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.767019 1169 tablet_bootstrap.cc:654] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.768326 1171 ts_tablet_manager.cc:1428] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.019s user 0.002s sys 0.000s
I20250905 19:52:36.768649 1171 tablet_bootstrap.cc:492] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.770408 2422 raft_consensus.cc:2747] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.664582 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet f48f9f789b6e46ec871a0fd007770dfc (table default.Table_2af22804e18e430ea738862d0bade226 [id=82e44ec5bc0e43e2ab11552c4127f225]) successfully deleted
W20250905 19:52:36.772727 1484 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 108171us
I20250905 19:52:36.773109 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]) successfully deleted
W20250905 19:52:36.664245 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6221d7d058944bdb9aa730791cf2417b because the tablet was not found. No further retry: Not found: Tablet not found: 6221d7d058944bdb9aa730791cf2417b
W20250905 19:52:36.773595 1472 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 109371us
I20250905 19:52:36.773695 2376 ts_tablet_manager.cc:1918] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.773797 2376 log.cc:1199] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9193487b4ab74dafa76a386ddcaf674b
W20250905 19:52:36.773984 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 5bd9380e8ffc432da4a684934c2df68f because the tablet was not found. No further retry: Not found: Tablet not found: 5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:36.774183 1171 tablet_bootstrap.cc:654] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.774377 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 68f2ccd8a7a64b89b44f2d6767ec3466 because the tablet was not found. No further retry: Not found: Tablet not found: 68f2ccd8a7a64b89b44f2d6767ec3466
W20250905 19:52:36.774650 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 98e5ca7bd3e84916a11e01afbf48171d because the tablet was not found. No further retry: Not found: Tablet not found: 98e5ca7bd3e84916a11e01afbf48171d
W20250905 19:52:36.774901 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff because the tablet was not found. No further retry: Not found: Tablet not found: c0ba1d4a7cd444c688d5a9bc8fd1a0ff
W20250905 19:52:36.775151 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet b13d282e1dc54d1faa2882c9b39d17cb because the tablet was not found. No further retry: Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb
W20250905 19:52:36.775373 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet b0702063ec1c4f6fba58e6d4f3912212 because the tablet was not found. No further retry: Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212
W20250905 19:52:36.775588 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 091ef981d6b54756830927ea111e78aa because the tablet was not found. No further retry: Not found: Tablet not found: 091ef981d6b54756830927ea111e78aa
I20250905 19:52:36.776502 2405 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" instance_seqno: 1757101933934195) as {username='slave'} at 127.0.104.3:45433; Asking this server to re-register.
W20250905 19:52:36.776587 849 leader_election.cc:343] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.776696 849 leader_election.cc:304] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.776961 2422 raft_consensus.cc:2747] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:36.777384 848 consensus_peers.cc:489] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:36.786161 1099 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:36.789008 767 raft_consensus.cc:3058] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:36.784780 849 leader_election.cc:343] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.789667 849 leader_election.cc:304] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:36.790098 849 leader_election.cc:343] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.799889 2652 ts_tablet_manager.cc:1918] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.2
I20250905 19:52:36.800120 2652 log.cc:1199] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/98e5ca7bd3e84916a11e01afbf48171d
I20250905 19:52:36.800674 2652 ts_tablet_manager.cc:1939] T 98e5ca7bd3e84916a11e01afbf48171d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:36.801551 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 78) took 5184 ms. Trace:
I20250905 19:52:36.801806 2652 rpcz_store.cc:276] 0905 19:52:31.616817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.616939 (+ 122us) service_pool.cc:225] Handling call
0905 19:52:36.552731 (+4935792us) ts_tablet_manager.cc:1145] Deleting tablet 98e5ca7bd3e84916a11e01afbf48171d
0905 19:52:36.552743 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.679410 (+126667us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.799766 (+120356us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.801528 (+ 1762us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4925502}
I20250905 19:52:36.790288 2413 raft_consensus.cc:2747] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.796309 2376 ts_tablet_manager.cc:1939] T 9193487b4ab74dafa76a386ddcaf674b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.807883 849 leader_election.cc:304] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.808506 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 100) took 7160 ms. Trace:
I20250905 19:52:36.811358 2376 rpcz_store.cc:276] 0905 19:52:29.648130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:29.648266 (+ 136us) service_pool.cc:225] Handling call
0905 19:52:36.726469 (+7078203us) ts_tablet_manager.cc:1145] Deleting tablet 9193487b4ab74dafa76a386ddcaf674b
0905 19:52:36.726477 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.770736 (+ 44259us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.773597 (+ 2861us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.808485 (+ 34888us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":7078010}
I20250905 19:52:36.811717 2376 tablet_replica.cc:331] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.811905 2376 raft_consensus.cc:2241] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.812073 2376 raft_consensus.cc:2270] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.812376 2422 raft_consensus.cc:2747] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.812620 2376 ts_tablet_manager.cc:1905] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.811934 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9193487b4ab74dafa76a386ddcaf674b (table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]) successfully deleted
W20250905 19:52:36.813207 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 44769f1170354870ae2d34fdb67ffbae because the tablet was not found. No further retry: Not found: Tablet not found: 44769f1170354870ae2d34fdb67ffbae
W20250905 19:52:36.813520 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet c46ea498d6a14d1ebdffea415cfc7114 because the tablet was not found. No further retry: Not found: Tablet not found: c46ea498d6a14d1ebdffea415cfc7114
W20250905 19:52:36.813755 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8503f6918ffb44efa36214633c93e31d because the tablet was not found. No further retry: Not found: Tablet not found: 8503f6918ffb44efa36214633c93e31d
W20250905 19:52:36.814003 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd793bce71e344f18bb177a53ab1bfc1 because the tablet was not found. No further retry: Not found: Tablet not found: fd793bce71e344f18bb177a53ab1bfc1
W20250905 19:52:36.814210 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d296428b70f9411190b6c408d81639f2 because the tablet was not found. No further retry: Not found: Tablet not found: d296428b70f9411190b6c408d81639f2
W20250905 19:52:36.814451 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6d12c7fbb84849c68274810b3fdcba04 because the tablet was not found. No further retry: Not found: Tablet not found: 6d12c7fbb84849c68274810b3fdcba04
W20250905 19:52:36.814700 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 43111a3512fb4fc0bdf4ee245b16695b because the tablet was not found. No further retry: Not found: Tablet not found: 43111a3512fb4fc0bdf4ee245b16695b
W20250905 19:52:36.814942 1472 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 8503f6918ffb44efa36214633c93e31d because the tablet was not found. No further retry: Not found: Tablet not found: 8503f6918ffb44efa36214633c93e31d
I20250905 19:52:36.798698 1171 tablet_bootstrap.cc:492] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.816092 1171 ts_tablet_manager.cc:1397] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.048s user 0.003s sys 0.000s
I20250905 19:52:36.821895 1171 raft_consensus.cc:357] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.822114 1171 raft_consensus.cc:383] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.822175 1171 raft_consensus.cc:738] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.822381 1171 consensus_queue.cc:260] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.822906 1171 ts_tablet_manager.cc:1428] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.001s sys 0.000s
I20250905 19:52:36.823189 1171 tablet_bootstrap.cc:492] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.826205 1171 tablet_bootstrap.cc:654] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.827391 849 leader_election.cc:343] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.827739 849 leader_election.cc:304] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.828052 2422 raft_consensus.cc:2747] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.832594 770 raft_consensus.cc:3058] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.835942 771 raft_consensus.cc:3058] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.847118 1170 tablet_bootstrap.cc:492] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:36.852375 2376 ts_tablet_manager.cc:1918] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:36.852522 2376 log.cc:1199] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9db56577a26e407b97fc33158c2c466c
I20250905 19:52:36.852816 2376 ts_tablet_manager.cc:1939] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.853250 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 112) took 5358 ms. Trace:
I20250905 19:52:36.853344 2376 rpcz_store.cc:276] 0905 19:52:31.495095 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.495145 (+ 50us) service_pool.cc:225] Handling call
0905 19:52:36.811660 (+5316515us) ts_tablet_manager.cc:1145] Deleting tablet 9db56577a26e407b97fc33158c2c466c
0905 19:52:36.811671 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.815602 (+ 3931us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.852258 (+ 36656us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.853231 (+ 973us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5316225}
I20250905 19:52:36.853672 2376 tablet_replica.cc:331] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:36.853854 2376 raft_consensus.cc:2241] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.853996 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9db56577a26e407b97fc33158c2c466c (table default.Table_d66d859b633f453393a574fa8885329b [id=ef8296fde71945bd9bc3c79fc9519dd7]) successfully deleted
I20250905 19:52:36.854126 2376 raft_consensus.cc:2270] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.854699 2376 ts_tablet_manager.cc:1905] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.854712 1170 ts_tablet_manager.cc:1397] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.196s user 0.002s sys 0.002s
I20250905 19:52:36.855885 1170 raft_consensus.cc:357] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.857681 1170 raft_consensus.cc:383] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.860056 2413 consensus_queue.cc:1035] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:36.861668 3101 catalog_manager.cc:1576] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: acquiring CA information for follower catalog manager: success
I20250905 19:52:36.864565 2403 master_service.cc:432] Got heartbeat from unknown tserver (permanent_uuid: "835daf30e4324782848290be181ae78f" instance_seqno: 1757101933433043) as {username='slave'} at 127.0.104.2:55401; Asking this server to re-register.
I20250905 19:52:36.865900 2405 ts_manager.cc:194] Registered new tserver with Master: 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:36.867146 849 leader_election.cc:343] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.867295 849 leader_election.cc:304] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.867520 2413 raft_consensus.cc:2747] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:36.868286 849 leader_election.cc:343] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: a1b4989f17454aa58c05727b797c2009
I20250905 19:52:36.868391 849 leader_election.cc:304] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:36.868538 849 raft_consensus.cc:2684] T a1b4989f17454aa58c05727b797c2009 P 835daf30e4324782848290be181ae78f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:36.870844 1171 tablet_bootstrap.cc:492] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.870971 1171 ts_tablet_manager.cc:1397] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.004s
I20250905 19:52:36.871665 1171 raft_consensus.cc:357] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.871834 1171 raft_consensus.cc:383] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.871891 1171 raft_consensus.cc:738] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.872093 1171 consensus_queue.cc:260] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.872598 1171 ts_tablet_manager.cc:1428] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:36.872817 1170 raft_consensus.cc:738] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.872872 1171 tablet_bootstrap.cc:492] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.874507 1171 tablet_bootstrap.cc:654] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.877940 960 heartbeater.cc:461] Registering TS with master...
I20250905 19:52:36.873060 1170 consensus_queue.cc:260] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.879563 1170 ts_tablet_manager.cc:1428] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.024s user 0.001s sys 0.000s
I20250905 19:52:36.879881 1170 tablet_bootstrap.cc:492] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:36.847694 1169 tablet_bootstrap.cc:492] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:36.883337 1169 ts_tablet_manager.cc:1397] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.228s user 0.002s sys 0.002s
I20250905 19:52:36.883414 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 98e5ca7bd3e84916a11e01afbf48171d (table default.Table_e5231e0b0e60414cb6b620f9f1f8a7ff [id=8f425647eea7481181ba823eb955d1e1]) successfully deleted
I20250905 19:52:36.884449 1169 raft_consensus.cc:357] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.884680 1169 raft_consensus.cc:383] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.884747 1169 raft_consensus.cc:738] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.887426 2930 raft_consensus.cc:491] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.887665 2930 raft_consensus.cc:513] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.889161 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14d5eb8bec074d2c8766fe7b159031e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:36.889668 984 leader_election.cc:343] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.889865 2930 leader_election.cc:290] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.890242 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14d5eb8bec074d2c8766fe7b159031e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.890565 907 raft_consensus.cc:2466] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:36.890959 986 leader_election.cc:304] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.891467 2930 raft_consensus.cc:2802] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.891588 2930 raft_consensus.cc:491] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.891721 2930 raft_consensus.cc:3058] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.885025 1169 consensus_queue.cc:260] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.901335 1169 ts_tablet_manager.cc:1428] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:36.904378 2652 tablet_replica.cc:331] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:36.904613 2652 raft_consensus.cc:2241] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:36.904680 3101 catalog_manager.cc:1604] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20250905 19:52:36.904913 2652 raft_consensus.cc:2270] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.905504 2652 ts_tablet_manager.cc:1905] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.907513 2376 ts_tablet_manager.cc:1918] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:36.907635 2376 log.cc:1199] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/c14dc571b586405f8fd1eaa452998978
I20250905 19:52:36.909962 1171 tablet_bootstrap.cc:492] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.910116 1171 ts_tablet_manager.cc:1397] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.008s
I20250905 19:52:36.910888 1171 raft_consensus.cc:357] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.911072 1171 raft_consensus.cc:383] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.911140 1171 raft_consensus.cc:738] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.911357 1171 consensus_queue.cc:260] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.911886 1171 ts_tablet_manager.cc:1428] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:36.912191 1171 tablet_bootstrap.cc:492] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.913862 1171 tablet_bootstrap.cc:654] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.918628 1169 tablet_bootstrap.cc:492] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:36.920181 1169 tablet_bootstrap.cc:654] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:36.921202 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 51999.6us
I20250905 19:52:36.923691 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5302b79208494e48a5f2778c5f52ad01" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.924015 2376 ts_tablet_manager.cc:1939] T c14dc571b586405f8fd1eaa452998978 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:36.924860 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 168) took 2914 ms. Trace:
I20250905 19:52:36.924953 2376 rpcz_store.cc:276] 0905 19:52:34.010538 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.010838 (+ 300us) service_pool.cc:225] Handling call
0905 19:52:36.853623 (+2842785us) ts_tablet_manager.cc:1145] Deleting tablet c14dc571b586405f8fd1eaa452998978
0905 19:52:36.853633 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.891483 (+ 37850us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.907394 (+ 15911us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:36.924840 (+ 17446us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2842533}
I20250905 19:52:36.925323 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 170) took 2914 ms. Trace:
I20250905 19:52:36.925400 2376 rpcz_store.cc:276] 0905 19:52:34.010753 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.011121 (+ 368us) service_pool.cc:225] Handling call
0905 19:52:36.925225 (+2914104us) ts_tablet_manager.cc:1145] Deleting tablet 578a6b9224cc4d8585d6c4b241762e1b
0905 19:52:36.925236 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.925309 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2913988}
I20250905 19:52:36.925488 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet c14dc571b586405f8fd1eaa452998978 (table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]) successfully deleted
I20250905 19:52:36.925697 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 174) took 2914 ms. Trace:
I20250905 19:52:36.925760 2376 rpcz_store.cc:276] 0905 19:52:34.011175 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.011256 (+ 81us) service_pool.cc:225] Handling call
0905 19:52:36.925623 (+2914367us) ts_tablet_manager.cc:1145] Deleting tablet 8ed6109bdf8f4caf8c223a42401b94ab
0905 19:52:36.925631 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.925683 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2914238}
W20250905 19:52:36.925940 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because the tablet was not found. No further retry: Not found: Tablet not found: 578a6b9224cc4d8585d6c4b241762e1b
I20250905 19:52:36.926023 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 175) took 2914 ms. Trace:
W20250905 19:52:36.926524 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8ed6109bdf8f4caf8c223a42401b94ab because the tablet was not found. No further retry: Not found: Tablet not found: 8ed6109bdf8f4caf8c223a42401b94ab
I20250905 19:52:36.926077 2376 rpcz_store.cc:276] 0905 19:52:34.011266 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.011406 (+ 140us) service_pool.cc:225] Handling call
0905 19:52:36.925955 (+2914549us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:36.925964 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.926010 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2914396}
I20250905 19:52:36.926934 2376 tablet_replica.cc:331] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:36.927089 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a8b6823580cb4040b49ed0a49b748965 because the tablet was not found. No further retry: Not found: Tablet not found: a8b6823580cb4040b49ed0a49b748965
I20250905 19:52:36.927136 2376 raft_consensus.cc:2241] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:36.927366 2376 pending_rounds.cc:70] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:36.927418 2376 pending_rounds.cc:77] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089608224530432 op_type: NO_OP noop_request { }
I20250905 19:52:36.927515 2376 raft_consensus.cc:2887] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:36.927597 2376 raft_consensus.cc:2270] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:36.928129 2376 ts_tablet_manager.cc:1905] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:36.933310 1170 tablet_bootstrap.cc:654] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.933629 1171 tablet_bootstrap.cc:492] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.933779 1171 ts_tablet_manager.cc:1397] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.001s
I20250905 19:52:36.934556 1171 raft_consensus.cc:357] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.934707 1171 raft_consensus.cc:383] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.934751 1171 raft_consensus.cc:738] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.934934 1171 consensus_queue.cc:260] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.935387 1171 ts_tablet_manager.cc:1428] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:36.942123 2931 consensus_queue.cc:1035] T e7086c5bfa4644d08e4f55231b85acf4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:36.948882 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 48954.5us
I20250905 19:52:36.951603 712 leader_election.cc:304] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.952522 3063 raft_consensus.cc:2747] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.952903 1171 tablet_bootstrap.cc:492] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.953979 712 leader_election.cc:304] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.954366 3063 raft_consensus.cc:2747] T a311f9b943d143a9a52d7771ce7186d7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.954787 1171 tablet_bootstrap.cc:654] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.956081 712 leader_election.cc:304] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.956768 3063 raft_consensus.cc:2747] T db098a883a61474fb06cb166041ee3e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.957372 712 leader_election.cc:304] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.957690 3063 raft_consensus.cc:2747] T cd5312852fe345d98039aeea241b7fe8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.958774 712 leader_election.cc:304] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.959077 3063 raft_consensus.cc:3058] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.961714 3063 raft_consensus.cc:2747] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.962746 712 leader_election.cc:304] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:36.963168 3063 raft_consensus.cc:2747] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:36.963861 2931 raft_consensus.cc:491] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.963979 2931 raft_consensus.cc:513] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.965158 2931 leader_election.cc:290] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.965759 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:36.966475 984 leader_election.cc:343] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.966759 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:36.966969 907 raft_consensus.cc:2466] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:36.967398 986 leader_election.cc:304] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.967866 2931 raft_consensus.cc:2802] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:36.968007 2931 raft_consensus.cc:491] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:36.968180 2931 raft_consensus.cc:3058] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.968441 1536 catalog_manager.cc:5582] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:36.969162 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:36.969497 1536 catalog_manager.cc:5582] T 48be704b238e481089c727b3e1b96a2f P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:36.969928 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:36.973347 2931 raft_consensus.cc:513] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.976156 1035 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:36.976238 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a860ab3f5ce54bd2b75bb71730a3c6e6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.976323 1035 raft_consensus.cc:3058] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.977044 1051 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95b32fff7d4745c79185a5d894ef3342" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.977186 1051 raft_consensus.cc:2466] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:36.978932 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "95b32fff7d4745c79185a5d894ef3342" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:36.979072 1041 raft_consensus.cc:3058] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.981386 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:36.981626 1041 raft_consensus.cc:2466] T 95b32fff7d4745c79185a5d894ef3342 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:36.963649 2930 raft_consensus.cc:513] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.979059 1035 raft_consensus.cc:2466] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:36.982544 2930 leader_election.cc:290] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:36.983290 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:36.984345 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:36.986795 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14d5eb8bec074d2c8766fe7b159031e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:36.987025 906 raft_consensus.cc:3058] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:36.988952 1171 tablet_bootstrap.cc:492] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:36.989089 1171 ts_tablet_manager.cc:1397] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.053s user 0.002s sys 0.000s
I20250905 19:52:36.989423 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "14d5eb8bec074d2c8766fe7b159031e5" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:36.989852 1171 raft_consensus.cc:357] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.990033 1171 raft_consensus.cc:383] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.990087 1171 raft_consensus.cc:738] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.990293 1171 consensus_queue.cc:260] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:36.990749 984 leader_election.cc:343] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:36.991441 1171 ts_tablet_manager.cc:1428] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:36.991787 1171 tablet_bootstrap.cc:492] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:36.990322 906 raft_consensus.cc:2466] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:36.993746 1171 tablet_bootstrap.cc:654] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:36.993880 986 leader_election.cc:304] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:36.994481 2998 raft_consensus.cc:2802] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:36.994613 2998 raft_consensus.cc:695] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:36.994871 2998 consensus_queue.cc:237] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.995148 1169 tablet_bootstrap.cc:492] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:36.995260 1169 ts_tablet_manager.cc:1397] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.077s user 0.002s sys 0.000s
I20250905 19:52:36.995682 2789 raft_consensus.cc:491] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.995797 2789 raft_consensus.cc:513] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.996302 2789 leader_election.cc:290] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.996407 2789 raft_consensus.cc:491] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.996496 2789 raft_consensus.cc:513] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.996891 2789 leader_election.cc:290] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.996984 2789 raft_consensus.cc:491] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.997056 2789 raft_consensus.cc:513] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.997429 2789 leader_election.cc:290] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.997526 2789 raft_consensus.cc:491] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:36.997597 2789 raft_consensus.cc:513] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.997956 2789 leader_election.cc:290] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:36.998234 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:36.999044 1169 raft_consensus.cc:357] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.999199 1169 raft_consensus.cc:383] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:36.999249 1169 raft_consensus.cc:738] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:36.999408 1169 consensus_queue.cc:260] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:36.999903 1169 ts_tablet_manager.cc:1428] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:37.000226 1169 tablet_bootstrap.cc:492] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.001770 1169 tablet_bootstrap.cc:654] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.002267 712 leader_election.cc:304] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.002719 712 leader_election.cc:304] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:37.002871 712 raft_consensus.cc:2684] T 3609de83732d4ad8857372acfbb5c32a P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:37.003019 1170 tablet_bootstrap.cc:492] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:37.003126 1170 ts_tablet_manager.cc:1397] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.123s user 0.003s sys 0.000s
I20250905 19:52:37.003510 712 leader_election.cc:304] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.003775 1849 raft_consensus.cc:2747] T 98e4e6d274574c0580315dcc2323c3d6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.003801 1170 raft_consensus.cc:357] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.003934 1170 raft_consensus.cc:383] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.003933 1849 raft_consensus.cc:2747] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.004024 1170 raft_consensus.cc:738] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.004199 1170 consensus_queue.cc:260] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.004737 1170 ts_tablet_manager.cc:1428] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:37.010035 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b53259c71d5443ebb12827ee78038fe" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.017634 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6597c91bec4f4143ab3ed40afe53a35e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.021647 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "48be704b238e481089c727b3e1b96a2f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.024051 2738 raft_consensus.cc:491] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.024220 2738 raft_consensus.cc:513] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.024892 2738 leader_election.cc:290] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.025454 849 leader_election.cc:304] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.026032 3155 raft_consensus.cc:2747] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.026199 849 leader_election.cc:343] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9db56577a26e407b97fc33158c2c466c
I20250905 19:52:37.029584 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:37.029866 849 leader_election.cc:304] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.003160 2652 ts_tablet_manager.cc:1918] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250905 19:52:37.030627 849 raft_consensus.cc:2684] T 9db56577a26e407b97fc33158c2c466c P 835daf30e4324782848290be181ae78f: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:37.030810 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:37.031033 2652 log.cc:1199] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/fd1280fefc3e43db9928567a734644f1
I20250905 19:52:37.031230 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:37.027320 2376 ts_tablet_manager.cc:1918] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.031500 2376 log.cc:1199] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/3609de83732d4ad8857372acfbb5c32a
I20250905 19:52:37.031912 2376 ts_tablet_manager.cc:1939] T 3609de83732d4ad8857372acfbb5c32a P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:37.032312 2652 ts_tablet_manager.cc:1939] T fd1280fefc3e43db9928567a734644f1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.032291 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 169) took 3021 ms. Trace:
I20250905 19:52:37.032383 2376 rpcz_store.cc:276] 0905 19:52:34.010661 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.011434 (+ 773us) service_pool.cc:225] Handling call
0905 19:52:36.926891 (+2915457us) ts_tablet_manager.cc:1145] Deleting tablet 3609de83732d4ad8857372acfbb5c32a
0905 19:52:36.926900 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.971430 (+ 44530us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.027207 (+ 55777us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.032270 (+ 5063us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2915231}
I20250905 19:52:37.032773 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 171) took 3021 ms. Trace:
I20250905 19:52:37.032855 2376 rpcz_store.cc:276] 0905 19:52:34.010887 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.011695 (+ 808us) service_pool.cc:225] Handling call
0905 19:52:37.032678 (+3020983us) ts_tablet_manager.cc:1145] Deleting tablet 1f1a8e7059b348cba34069ef49bf1aad
0905 19:52:37.032689 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.032759 (+ 70us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3020836}
I20250905 19:52:37.033170 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 173) took 3022 ms. Trace:
I20250905 19:52:37.033248 2376 rpcz_store.cc:276] 0905 19:52:34.011074 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.012032 (+ 958us) service_pool.cc:225] Handling call
0905 19:52:37.033087 (+3021055us) ts_tablet_manager.cc:1145] Deleting tablet 98e5ca7bd3e84916a11e01afbf48171d
0905 19:52:37.033096 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.033154 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3020913}
I20250905 19:52:37.033569 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 176) took 3022 ms. Trace:
I20250905 19:52:37.033666 2376 rpcz_store.cc:276] 0905 19:52:34.011361 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.012203 (+ 842us) service_pool.cc:225] Handling call
0905 19:52:37.033494 (+3021291us) ts_tablet_manager.cc:1145] Deleting tablet 42f3c1efb94e446fb0128b6b412821e2
0905 19:52:37.033501 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.033553 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3021128}
I20250905 19:52:37.033928 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 182) took 3019 ms. Trace:
I20250905 19:52:37.034004 2376 rpcz_store.cc:276] 0905 19:52:34.014485 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.014533 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:37.033863 (+3019330us) ts_tablet_manager.cc:1145] Deleting tablet 00d8144bb4f14358af539b7aae326918
0905 19:52:37.033870 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.033915 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3019150}
I20250905 19:52:37.034277 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 185) took 3019 ms. Trace:
I20250905 19:52:37.034373 2376 rpcz_store.cc:276] 0905 19:52:34.014810 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.014949 (+ 139us) service_pool.cc:225] Handling call
0905 19:52:37.034205 (+3019256us) ts_tablet_manager.cc:1145] Deleting tablet 05ae83eea5f74e9187713e8b82c9a23b
0905 19:52:37.034213 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.034261 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3019114}
I20250905 19:52:37.034693 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 184) took 3019 ms. Trace:
I20250905 19:52:37.034788 2376 rpcz_store.cc:276] 0905 19:52:34.014724 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.015036 (+ 312us) service_pool.cc:225] Handling call
0905 19:52:37.034622 (+3019586us) ts_tablet_manager.cc:1145] Deleting tablet 0267ac3e3f1a45978807737eca5d91ef
0905 19:52:37.034630 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.034677 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3019385}
I20250905 19:52:37.035094 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 186) took 3020 ms. Trace:
I20250905 19:52:37.035188 2376 rpcz_store.cc:276] 0905 19:52:34.014898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.015135 (+ 237us) service_pool.cc:225] Handling call
0905 19:52:37.035027 (+3019892us) ts_tablet_manager.cc:1145] Deleting tablet 8edf4b54974c427a855eee5792d75292
0905 19:52:37.035034 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.035079 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3019766}
I20250905 19:52:37.035519 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 179) took 3021 ms. Trace:
I20250905 19:52:37.035612 2376 rpcz_store.cc:276] 0905 19:52:34.014205 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.015286 (+ 1081us) service_pool.cc:225] Handling call
0905 19:52:37.035422 (+3020136us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:37.035430 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.035504 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3019989}
I20250905 19:52:37.036403 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 81) took 5414 ms. Trace:
W20250905 19:52:37.036747 849 leader_election.cc:343] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.036700 3155 consensus_queue.cc:1035] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.037153 849 leader_election.cc:304] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.037549 2738 raft_consensus.cc:2747] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.038357 2652 rpcz_store.cc:276] 0905 19:52:31.622329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.622849 (+ 520us) service_pool.cc:225] Handling call
0905 19:52:36.883184 (+5260335us) ts_tablet_manager.cc:1145] Deleting tablet fd1280fefc3e43db9928567a734644f1
0905 19:52:36.904323 (+ 21139us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:36.985701 (+ 81378us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.003041 (+ 17340us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.036380 (+ 33339us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5255118}
I20250905 19:52:37.039146 2652 tablet_replica.cc:331] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.040803 2652 raft_consensus.cc:2241] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.042876 2652 raft_consensus.cc:2270] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.044011 1170 tablet_bootstrap.cc:492] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:37.045858 1170 tablet_bootstrap.cc:654] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.048700 2652 ts_tablet_manager.cc:1905] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.052817 849 leader_election.cc:304] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.054021 712 leader_election.cc:304] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.054533 712 leader_election.cc:304] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.054792 1849 raft_consensus.cc:2747] T 3b53259c71d5443ebb12827ee78038fe P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.054841 2738 raft_consensus.cc:2802] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.054941 1849 raft_consensus.cc:2747] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.054970 2738 raft_consensus.cc:695] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:37.055281 712 leader_election.cc:304] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.055244 2738 consensus_queue.cc:237] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.055470 1849 raft_consensus.cc:2747] T 48be704b238e481089c727b3e1b96a2f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.054248 1171 tablet_bootstrap.cc:492] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.056799 1171 ts_tablet_manager.cc:1397] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.065s user 0.004s sys 0.000s
I20250905 19:52:37.057670 1171 raft_consensus.cc:357] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.057884 1171 raft_consensus.cc:383] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.057956 1171 raft_consensus.cc:738] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.058174 1171 consensus_queue.cc:260] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.058826 1171 ts_tablet_manager.cc:1428] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:37.059160 1171 tablet_bootstrap.cc:492] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.032919 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 3609de83732d4ad8857372acfbb5c32a (table default.Table_f6e643730c10465cb8585cb48c1c5e04 [id=fa983711b6574ebeb9dfb2cca302bdc1]) successfully deleted
W20250905 19:52:37.064432 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because the tablet was not found. No further retry: Not found: Tablet not found: 1f1a8e7059b348cba34069ef49bf1aad
W20250905 19:52:37.064770 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 98e5ca7bd3e84916a11e01afbf48171d because the tablet was not found. No further retry: Not found: Tablet not found: 98e5ca7bd3e84916a11e01afbf48171d
W20250905 19:52:37.064997 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 42f3c1efb94e446fb0128b6b412821e2 because the tablet was not found. No further retry: Not found: Tablet not found: 42f3c1efb94e446fb0128b6b412821e2
W20250905 19:52:37.065198 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 00d8144bb4f14358af539b7aae326918 because the tablet was not found. No further retry: Not found: Tablet not found: 00d8144bb4f14358af539b7aae326918
W20250905 19:52:37.065436 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 05ae83eea5f74e9187713e8b82c9a23b because the tablet was not found. No further retry: Not found: Tablet not found: 05ae83eea5f74e9187713e8b82c9a23b
W20250905 19:52:37.065666 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 0267ac3e3f1a45978807737eca5d91ef because the tablet was not found. No further retry: Not found: Tablet not found: 0267ac3e3f1a45978807737eca5d91ef
W20250905 19:52:37.065886 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8edf4b54974c427a855eee5792d75292 because the tablet was not found. No further retry: Not found: Tablet not found: 8edf4b54974c427a855eee5792d75292
W20250905 19:52:37.066082 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 06850faa25de4b3d83f6bf98ccd370f0 because the tablet was not found. No further retry: Not found: Tablet not found: 06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:37.039340 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet fd1280fefc3e43db9928567a734644f1 (table default.Table_c8afd57df21440b5923aef4dd927ae45 [id=4447588b8d2443e5a561758689326e26]) successfully deleted
I20250905 19:52:37.067137 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:37.067693 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:37.068086 1536 catalog_manager.cc:5582] T 8e0c478efe044ad0867e47f49374a66b P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:37.069144 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:37.071795 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180437d1534946619384307ddcb7c937" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.072490 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "160a7814abf9423dbe965c252c20fa7f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.072683 1045 raft_consensus.cc:2466] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:37.073026 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "862214a31059407baefef8a390aaea69" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.073218 1045 raft_consensus.cc:2466] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:37.073606 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.073727 1045 raft_consensus.cc:2466] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:37.075345 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.075491 1045 raft_consensus.cc:2466] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:37.035923 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 180) took 3021 ms. Trace:
I20250905 19:52:37.079466 2376 rpcz_store.cc:276] 0905 19:52:34.014293 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.015602 (+ 1309us) service_pool.cc:225] Handling call
0905 19:52:37.035853 (+3020251us) ts_tablet_manager.cc:1145] Deleting tablet 53ac46ce05694bc2b53e195638be9e23
0905 19:52:37.035860 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.035907 (+ 47us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3020084}
I20250905 19:52:37.080214 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 181) took 3065 ms. Trace:
I20250905 19:52:37.080309 2376 rpcz_store.cc:276] 0905 19:52:34.014373 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.016010 (+ 1637us) service_pool.cc:225] Handling call
0905 19:52:37.079842 (+3063832us) ts_tablet_manager.cc:1145] Deleting tablet 78076f6534814939ae8c2db9a4bd1879
0905 19:52:37.080108 (+ 266us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.080194 (+ 86us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3063638}
I20250905 19:52:37.080673 2376 tablet_replica.cc:331] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:37.080919 2376 raft_consensus.cc:2241] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.081053 2376 raft_consensus.cc:2270] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.081703 2376 ts_tablet_manager.cc:1905] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:37.083114 849 leader_election.cc:343] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.083428 849 leader_election.cc:304] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.083750 2738 raft_consensus.cc:2802] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.083873 2738 raft_consensus.cc:491] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.084034 2738 raft_consensus.cc:3058] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.087324 1171 tablet_bootstrap.cc:654] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:37.080096 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 53ac46ce05694bc2b53e195638be9e23 because the tablet was not found. No further retry: Not found: Tablet not found: 53ac46ce05694bc2b53e195638be9e23
W20250905 19:52:37.089053 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 78076f6534814939ae8c2db9a4bd1879 because the tablet was not found. No further retry: Not found: Tablet not found: 78076f6534814939ae8c2db9a4bd1879
I20250905 19:52:37.069366 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:37.090306 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:37.090535 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:37.090826 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:37.091174 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:37.098419 1044 raft_consensus.cc:2391] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 835daf30e4324782848290be181ae78f in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:37.100147 849 leader_election.cc:304] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.100102 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:37.087203 2738 raft_consensus.cc:513] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:37.100924 984 leader_election.cc:343] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.101564 2903 raft_consensus.cc:2802] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.101725 2903 raft_consensus.cc:491] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.101872 2903 raft_consensus.cc:3058] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.105470 2903 raft_consensus.cc:513] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.108511 1536 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:37.105885 849 leader_election.cc:304] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.101064 2738 leader_election.cc:290] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.109359 2413 raft_consensus.cc:2802] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.109557 2413 raft_consensus.cc:491] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.109704 2413 raft_consensus.cc:3058] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.107764 1171 tablet_bootstrap.cc:492] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.110793 1171 ts_tablet_manager.cc:1397] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.052s user 0.002s sys 0.002s
I20250905 19:52:37.111711 1171 raft_consensus.cc:357] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.111865 1171 raft_consensus.cc:383] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.111922 1171 raft_consensus.cc:738] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.112133 1171 consensus_queue.cc:260] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.112353 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:37.112480 907 raft_consensus.cc:3058] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.112632 1171 ts_tablet_manager.cc:1428] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:37.112895 1171 tablet_bootstrap.cc:492] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.114482 1171 tablet_bootstrap.cc:654] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.115110 907 raft_consensus.cc:2466] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:37.115383 2413 raft_consensus.cc:513] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.115842 986 leader_election.cc:304] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.116170 3160 raft_consensus.cc:2802] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.116274 3160 raft_consensus.cc:695] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:37.116303 2413 leader_election.cc:290] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.116426 3063 raft_consensus.cc:491] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.116539 3160 consensus_queue.cc:237] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.116606 3063 raft_consensus.cc:513] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.117168 2422 consensus_queue.cc:1035] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.117408 3155 raft_consensus.cc:491] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.117293 3063 leader_election.cc:290] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.117534 3155 raft_consensus.cc:513] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.117967 3155 leader_election.cc:290] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.117969 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.118186 1044 raft_consensus.cc:2466] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:37.118753 711 leader_election.cc:304] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:37.119236 3063 raft_consensus.cc:2802] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.119403 3063 raft_consensus.cc:491] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.117970 2903 leader_election.cc:290] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.119596 3063 raft_consensus.cc:3058] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.119616 2422 consensus_queue.cc:1035] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.119839 849 leader_election.cc:304] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.120744 2903 raft_consensus.cc:2802] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.120869 2903 raft_consensus.cc:491] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.121001 2903 raft_consensus.cc:3058] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.122946 3063 raft_consensus.cc:513] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.124195 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:37.124400 1044 raft_consensus.cc:3058] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.127312 1044 raft_consensus.cc:2466] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:37.127861 711 leader_election.cc:304] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:37.128206 2738 raft_consensus.cc:491] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.128255 1849 raft_consensus.cc:2802] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.128334 2738 raft_consensus.cc:513] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.129660 2738 leader_election.cc:290] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.132519 2903 raft_consensus.cc:513] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.134868 2652 ts_tablet_manager.cc:1918] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.136770 2931 leader_election.cc:290] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.145375 1849 raft_consensus.cc:695] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:37.146030 1849 consensus_queue.cc:237] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.147217 2652 log.cc:1199] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:37.136103 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.136651 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:37.148350 3063 leader_election.cc:290] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.148520 2652 ts_tablet_manager.cc:1939] T baefaa1f042348c1874e306f1e5a8781 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.148912 907 raft_consensus.cc:2391] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:37.150044 2903 leader_election.cc:290] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.156051 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180437d1534946619384307ddcb7c937" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.159718 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "160a7814abf9423dbe965c252c20fa7f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.161978 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "862214a31059407baefef8a390aaea69" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.163908 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 84) took 5478 ms. Trace:
I20250905 19:52:37.164109 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "160a7814abf9423dbe965c252c20fa7f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:37.164772 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:37.165310 1035 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "862214a31059407baefef8a390aaea69" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:37.165299 2652 rpcz_store.cc:276] 0905 19:52:31.685334 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.685462 (+ 128us) service_pool.cc:225] Handling call
0905 19:52:37.039086 (+5353624us) ts_tablet_manager.cc:1145] Deleting tablet baefaa1f042348c1874e306f1e5a8781
0905 19:52:37.039099 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.106096 (+ 66997us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.134725 (+ 28629us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.163877 (+ 29152us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5353419}
I20250905 19:52:37.165509 1035 raft_consensus.cc:3058] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.166345 2652 tablet_replica.cc:331] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.166801 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.166908 2652 raft_consensus.cc:2241] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.167490 2652 raft_consensus.cc:2270] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.168958 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet baefaa1f042348c1874e306f1e5a8781 (table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]) successfully deleted
I20250905 19:52:37.165513 1048 raft_consensus.cc:3058] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:37.170933 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 9f5d19f0216a4f069237e48c62ee7875 consensus state has no leader
I20250905 19:52:37.168530 2652 ts_tablet_manager.cc:1905] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.165118 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a551e641a19f4d3f924fb176c4a25ca7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.173480 1048 raft_consensus.cc:2466] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:37.170322 1035 raft_consensus.cc:2466] T 862214a31059407baefef8a390aaea69 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:37.165623 1045 raft_consensus.cc:3058] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.177326 1045 raft_consensus.cc:2466] T 160a7814abf9423dbe965c252c20fa7f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:37.177873 1171 tablet_bootstrap.cc:492] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.178138 1171 ts_tablet_manager.cc:1397] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.065s user 0.003s sys 0.001s
I20250905 19:52:37.179000 1171 raft_consensus.cc:357] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.179316 1171 raft_consensus.cc:383] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.179440 1171 raft_consensus.cc:738] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.179849 1171 consensus_queue.cc:260] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.183362 1171 ts_tablet_manager.cc:1428] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20250905 19:52:37.184653 1171 tablet_bootstrap.cc:492] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.187013 1171 tablet_bootstrap.cc:654] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.187012 2652 ts_tablet_manager.cc:1918] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.193420 2403 ts_manager.cc:194] Registered new tserver with Master: 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.150377 2376 ts_tablet_manager.cc:1918] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:37.197490 2376 log.cc:1199] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/14d5eb8bec074d2c8766fe7b159031e5
I20250905 19:52:37.187642 2903 raft_consensus.cc:491] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.197904 2376 ts_tablet_manager.cc:1939] T 14d5eb8bec074d2c8766fe7b159031e5 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:37.197952 2903 raft_consensus.cc:513] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.198583 2903 leader_election.cc:290] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.198700 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 187) took 3181 ms. Trace:
I20250905 19:52:37.198799 2376 rpcz_store.cc:276] 0905 19:52:34.017375 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.017523 (+ 148us) service_pool.cc:225] Handling call
0905 19:52:37.080615 (+3063092us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:37.080627 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.126986 (+ 46359us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.150248 (+ 23262us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.198675 (+ 48427us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3062935}
I20250905 19:52:37.199538 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 14d5eb8bec074d2c8766fe7b159031e5 (table default.Table_b21bb0be9ae1435c8929de3f6b662d8c [id=a9a1d004ba9f4415b2fb958c789d66ba]) successfully deleted
I20250905 19:52:37.200590 2903 raft_consensus.cc:491] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.200802 2903 raft_consensus.cc:513] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.201436 2903 leader_election.cc:290] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.201798 2652 log.cc:1199] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/3de936d2309a411eabbd59f7b73ce7f3
I20250905 19:52:37.202452 2652 ts_tablet_manager.cc:1939] T 3de936d2309a411eabbd59f7b73ce7f3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.205106 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 85) took 5516 ms. Trace:
I20250905 19:52:37.205843 2652 rpcz_store.cc:276] 0905 19:52:31.688144 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.688309 (+ 165us) service_pool.cc:225] Handling call
0905 19:52:37.166232 (+5477923us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:37.166244 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.179917 (+ 13673us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.186846 (+ 6929us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.205063 (+ 18217us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5458210}
I20250905 19:52:37.206681 2652 tablet_replica.cc:331] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.207270 2652 raft_consensus.cc:2241] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.207304 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 3de936d2309a411eabbd59f7b73ce7f3 (table default.Table_88ff02c5dce2458784f09b64cee688f1 [id=378b42fcb4e741e4b4755deb4b0ef51a]) successfully deleted
I20250905 19:52:37.211151 2652 raft_consensus.cc:2270] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.212175 2652 ts_tablet_manager.cc:1905] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.223347 882 tablet_service.cc:1468] Processing CreateTablet for tablet 3b3bcd5a4f9c46e99664f6cbe9167301 (DEFAULT_TABLE table=default.Table_3c5a0a7b75424d2ab2975a941bab388b [id=c3c7d5d0e4394865a0488381d8cb4c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:37.223871 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3b3bcd5a4f9c46e99664f6cbe9167301 (DEFAULT_TABLE table=default.Table_3c5a0a7b75424d2ab2975a941bab388b [id=c3c7d5d0e4394865a0488381d8cb4c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:37.224433 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3bcd5a4f9c46e99664f6cbe9167301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:37.229118 2652 ts_tablet_manager.cc:1918] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.234298 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.235767 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "160a7814abf9423dbe965c252c20fa7f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:37.236450 2652 log.cc:1199] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9e9994863c374354838aa0f817d0057e
I20250905 19:52:37.237437 2652 ts_tablet_manager.cc:1939] T 9e9994863c374354838aa0f817d0057e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.238706 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 86) took 5516 ms. Trace:
I20250905 19:52:37.240007 1849 raft_consensus.cc:491] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.240245 1849 raft_consensus.cc:513] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.241405 1849 leader_election.cc:290] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.241499 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.239754 2652 rpcz_store.cc:276] 0905 19:52:31.722681 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.723053 (+ 372us) service_pool.cc:225] Handling call
0905 19:52:37.206601 (+5483548us) ts_tablet_manager.cc:1145] Deleting tablet 9e9994863c374354838aa0f817d0057e
0905 19:52:37.206616 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.225426 (+ 18810us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.228989 (+ 3563us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.238642 (+ 9653us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5481273}
I20250905 19:52:37.242743 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9e9994863c374354838aa0f817d0057e (table default.Table_56e29cf512764408b243635ae01a6ab0 [id=aad015c74608490985ee75fbd380f0e3]) successfully deleted
I20250905 19:52:37.242492 2652 tablet_replica.cc:331] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.244597 2652 raft_consensus.cc:2241] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.245167 2652 raft_consensus.cc:2270] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.246239 2652 ts_tablet_manager.cc:1905] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.251049 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:37.252089 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "862214a31059407baefef8a390aaea69" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:37.252993 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a551e641a19f4d3f924fb176c4a25ca7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.255836 2652 ts_tablet_manager.cc:1918] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.256336 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.258194 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:37.262708 2652 log.cc:1199] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/723168b075874dfcb3eb50c5134c499e
I20250905 19:52:37.263273 2652 ts_tablet_manager.cc:1939] T 723168b075874dfcb3eb50c5134c499e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.264302 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 88) took 5532 ms. Trace:
I20250905 19:52:37.264547 2652 rpcz_store.cc:276] 0905 19:52:31.731937 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.732867 (+ 930us) service_pool.cc:225] Handling call
0905 19:52:37.242394 (+5509527us) ts_tablet_manager.cc:1145] Deleting tablet 723168b075874dfcb3eb50c5134c499e
0905 19:52:37.242404 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.252062 (+ 9658us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.255696 (+ 3634us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.264242 (+ 8546us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5506340}
I20250905 19:52:37.265105 2652 tablet_replica.cc:331] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.265195 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 723168b075874dfcb3eb50c5134c499e (table default.Table_460fbdca606041adbd01239f6cfff523 [id=0b1ae52e28114d889c7438d8720940fc]) successfully deleted
I20250905 19:52:37.265652 2652 raft_consensus.cc:2241] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.265919 2652 raft_consensus.cc:2270] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.267580 2376 tablet_replica.cc:331] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:37.267786 2376 raft_consensus.cc:2241] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.267940 2376 raft_consensus.cc:2270] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.268230 2652 ts_tablet_manager.cc:1905] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.268364 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.268507 2376 ts_tablet_manager.cc:1905] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.273901 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "55cecf59d3f24db88bb10b8e12c12f82" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.274488 2376 ts_tablet_manager.cc:1918] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:37.274600 2376 log.cc:1199] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6ebab2e3698f4f60a770bde9cc019600
I20250905 19:52:37.274531 849 leader_election.cc:304] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.274902 2376 ts_tablet_manager.cc:1939] T 6ebab2e3698f4f60a770bde9cc019600 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:37.274955 2738 raft_consensus.cc:2747] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.276468 849 leader_election.cc:343] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.276769 849 leader_election.cc:304] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.277077 2413 raft_consensus.cc:2802] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.277204 2413 raft_consensus.cc:695] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:37.277468 2413 consensus_queue.cc:237] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.277941 2652 ts_tablet_manager.cc:1918] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.278100 2652 log.cc:1199] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/a3805127dd78489e8545da7414ead904
I20250905 19:52:37.278522 2652 ts_tablet_manager.cc:1939] T a3805127dd78489e8545da7414ead904 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.279589 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 87) took 5550 ms. Trace:
I20250905 19:52:37.279744 2652 rpcz_store.cc:276] 0905 19:52:31.731777 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.733519 (+ 1742us) service_pool.cc:225] Handling call
0905 19:52:37.265016 (+5531497us) ts_tablet_manager.cc:1145] Deleting tablet a3805127dd78489e8545da7414ead904
0905 19:52:37.265029 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.273274 (+ 8245us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.277762 (+ 4488us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.279527 (+ 1765us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5516777}
I20250905 19:52:37.280273 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 92) took 5506 ms. Trace:
I20250905 19:52:37.280433 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet a3805127dd78489e8545da7414ead904 (table default.Table_e65ca281d8804277a83ab7bfc2ab9cca [id=3ed6b530762e4290aa5c45ee1a55c100]) successfully deleted
I20250905 19:52:37.280416 2652 rpcz_store.cc:276] 0905 19:52:31.773527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.773588 (+ 61us) service_pool.cc:225] Handling call
0905 19:52:37.280139 (+5506551us) ts_tablet_manager.cc:1145] Deleting tablet 9db56577a26e407b97fc33158c2c466c
0905 19:52:37.280148 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.280221 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5506336}
I20250905 19:52:37.280830 2652 tablet_replica.cc:331] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
W20250905 19:52:37.280944 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 9db56577a26e407b97fc33158c2c466c because the tablet was not found. No further retry: Not found: Tablet not found: 9db56577a26e407b97fc33158c2c466c
I20250905 19:52:37.281134 2652 raft_consensus.cc:2241] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.282136 849 leader_election.cc:304] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.282450 2413 raft_consensus.cc:2802] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.282632 2413 raft_consensus.cc:695] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:37.282933 2413 consensus_queue.cc:237] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.283026 2652 raft_consensus.cc:2270] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.283732 2652 ts_tablet_manager.cc:1905] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.286619 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3bcd5a4f9c46e99664f6cbe9167301. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:37.287235 1472 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.62:38687, user_credentials={real_user=slave}} blocked reactor thread for 68038.4us
I20250905 19:52:37.288292 751 tablet_service.cc:1468] Processing CreateTablet for tablet 3b3bcd5a4f9c46e99664f6cbe9167301 (DEFAULT_TABLE table=default.Table_3c5a0a7b75424d2ab2975a941bab388b [id=c3c7d5d0e4394865a0488381d8cb4c97]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:37.288801 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3b3bcd5a4f9c46e99664f6cbe9167301. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:37.296589 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:37.299683 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 189) took 3281 ms. Trace:
I20250905 19:52:37.299790 2376 rpcz_store.cc:276] 0905 19:52:34.018491 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.018734 (+ 243us) service_pool.cc:225] Handling call
0905 19:52:37.199317 (+3180583us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:37.267498 (+ 68181us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.271670 (+ 4172us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.274409 (+ 2739us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.299657 (+ 25248us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3180347}
I20250905 19:52:37.300189 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 190) took 3281 ms. Trace:
I20250905 19:52:37.300268 2376 rpcz_store.cc:276] 0905 19:52:34.018595 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.018989 (+ 394us) service_pool.cc:225] Handling call
0905 19:52:37.300085 (+3281096us) ts_tablet_manager.cc:1145] Deleting tablet 2683a36998674f739b41d2a57339954c
0905 19:52:37.300095 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.300175 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3280923}
I20250905 19:52:37.300529 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 191) took 3266 ms. Trace:
I20250905 19:52:37.300587 2376 rpcz_store.cc:276] 0905 19:52:34.033951 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.034004 (+ 53us) service_pool.cc:225] Handling call
0905 19:52:37.300459 (+3266455us) ts_tablet_manager.cc:1145] Deleting tablet baefaa1f042348c1874e306f1e5a8781
0905 19:52:37.300467 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.300517 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3266235}
I20250905 19:52:37.300814 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 199) took 3259 ms. Trace:
I20250905 19:52:37.300868 2376 rpcz_store.cc:276] 0905 19:52:34.040938 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.040986 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:37.300750 (+3259764us) ts_tablet_manager.cc:1145] Deleting tablet ab4f69c45295422f85b310aa55220e9b
0905 19:52:37.300758 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.300804 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3259422}
I20250905 19:52:37.301107 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 201) took 3258 ms. Trace:
I20250905 19:52:37.301162 2376 rpcz_store.cc:276] 0905 19:52:34.042549 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.042603 (+ 54us) service_pool.cc:225] Handling call
0905 19:52:37.301025 (+3258422us) ts_tablet_manager.cc:1145] Deleting tablet 4d2ed1bc09c94526bc9662f2d4d21f8f
0905 19:52:37.301032 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.301096 (+ 64us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3258207}
I20250905 19:52:37.301416 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 204) took 3102 ms. Trace:
I20250905 19:52:37.301471 2376 rpcz_store.cc:276] 0905 19:52:34.199322 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.199583 (+ 261us) service_pool.cc:225] Handling call
0905 19:52:37.301333 (+3101750us) ts_tablet_manager.cc:1145] Deleting tablet a5af8232d31240ff855b79dcea29396b
0905 19:52:37.301341 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.301404 (+ 63us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3101520}
I20250905 19:52:37.301705 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 206) took 3101 ms. Trace:
I20250905 19:52:37.301764 2376 rpcz_store.cc:276] 0905 19:52:34.199835 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.199892 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:37.301632 (+3101740us) ts_tablet_manager.cc:1145] Deleting tablet f28e403dc2f6471ebea1805fd843cfb1
0905 19:52:37.301641 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.301693 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3101543}
I20250905 19:52:37.302018 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 207) took 3102 ms. Trace:
I20250905 19:52:37.302084 2376 rpcz_store.cc:276] 0905 19:52:34.200009 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.200065 (+ 56us) service_pool.cc:225] Handling call
0905 19:52:37.301949 (+3101884us) ts_tablet_manager.cc:1145] Deleting tablet b0702063ec1c4f6fba58e6d4f3912212
0905 19:52:37.301957 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.302006 (+ 49us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3101680}
I20250905 19:52:37.302351 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 208) took 3101 ms. Trace:
I20250905 19:52:37.302412 2376 rpcz_store.cc:276] 0905 19:52:34.200621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.200665 (+ 44us) service_pool.cc:225] Handling call
0905 19:52:37.302271 (+3101606us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:37.302279 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.302337 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3101437}
I20250905 19:52:37.302675 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 210) took 3086 ms. Trace:
I20250905 19:52:37.302739 2376 rpcz_store.cc:276] 0905 19:52:34.216281 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.216348 (+ 67us) service_pool.cc:225] Handling call
0905 19:52:37.302591 (+3086243us) ts_tablet_manager.cc:1145] Deleting tablet 00ac56c449ba4b7b8dbd5e87120799d9
0905 19:52:37.302599 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.302661 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3086016}
I20250905 19:52:37.302980 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 211) took 3086 ms. Trace:
I20250905 19:52:37.303052 2376 rpcz_store.cc:276] 0905 19:52:34.216401 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.216626 (+ 225us) service_pool.cc:225] Handling call
0905 19:52:37.302908 (+3086282us) ts_tablet_manager.cc:1145] Deleting tablet 03a12cf719f34096bb82f5846bbe3b2d
0905 19:52:37.302916 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.302967 (+ 51us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3086150}
I20250905 19:52:37.303342 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 214) took 3086 ms. Trace:
I20250905 19:52:37.303412 2376 rpcz_store.cc:276] 0905 19:52:34.217271 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.217329 (+ 58us) service_pool.cc:225] Handling call
0905 19:52:37.303233 (+3085904us) ts_tablet_manager.cc:1145] Deleting tablet b13d282e1dc54d1faa2882c9b39d17cb
0905 19:52:37.303255 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.303328 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085737}
I20250905 19:52:37.303478 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6ebab2e3698f4f60a770bde9cc019600 (table default.Table_68450db54c5641a1a249cdcf5fa2d5ec [id=94f2531a27494c32bcd549a25287867f]) successfully deleted
I20250905 19:52:37.303664 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 215) took 3085 ms. Trace:
I20250905 19:52:37.303720 2376 rpcz_store.cc:276] 0905 19:52:34.217712 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.217753 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:37.303595 (+3085842us) ts_tablet_manager.cc:1145] Deleting tablet 302053e1611b4e22a9a2e7b60ae68f27
0905 19:52:37.303602 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.303652 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085712}
W20250905 19:52:37.303782 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 2683a36998674f739b41d2a57339954c because the tablet was not found. No further retry: Not found: Tablet not found: 2683a36998674f739b41d2a57339954c
W20250905 19:52:37.304003 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet baefaa1f042348c1874e306f1e5a8781 because the tablet was not found. No further retry: Not found: Tablet not found: baefaa1f042348c1874e306f1e5a8781
I20250905 19:52:37.304018 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 216) took 3085 ms. Trace:
I20250905 19:52:37.304075 2376 rpcz_store.cc:276] 0905 19:52:34.218326 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.218384 (+ 58us) service_pool.cc:225] Handling call
0905 19:52:37.303905 (+3085521us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:37.303913 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.304003 (+ 90us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085385}
W20250905 19:52:37.304215 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ab4f69c45295422f85b310aa55220e9b because the tablet was not found. No further retry: Not found: Tablet not found: ab4f69c45295422f85b310aa55220e9b
I20250905 19:52:37.304353 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 217) took 3085 ms. Trace:
W20250905 19:52:37.304395 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 4d2ed1bc09c94526bc9662f2d4d21f8f because the tablet was not found. No further retry: Not found: Tablet not found: 4d2ed1bc09c94526bc9662f2d4d21f8f
I20250905 19:52:37.304437 2376 rpcz_store.cc:276] 0905 19:52:34.218701 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.218743 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:37.304282 (+3085539us) ts_tablet_manager.cc:1145] Deleting tablet 9630f1feb6d74472b5f6f4b54151311c
0905 19:52:37.304290 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.304340 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085397}
W20250905 19:52:37.304596 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a5af8232d31240ff855b79dcea29396b because the tablet was not found. No further retry: Not found: Tablet not found: a5af8232d31240ff855b79dcea29396b
I20250905 19:52:37.304699 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 218) took 3085 ms. Trace:
W20250905 19:52:37.304774 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet f28e403dc2f6471ebea1805fd843cfb1 because the tablet was not found. No further retry: Not found: Tablet not found: f28e403dc2f6471ebea1805fd843cfb1
I20250905 19:52:37.304756 2376 rpcz_store.cc:276] 0905 19:52:34.219044 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.219083 (+ 39us) service_pool.cc:225] Handling call
0905 19:52:37.304633 (+3085550us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:37.304641 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.304686 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085428}
W20250905 19:52:37.304955 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet b0702063ec1c4f6fba58e6d4f3912212 because the tablet was not found. No further retry: Not found: Tablet not found: b0702063ec1c4f6fba58e6d4f3912212
I20250905 19:52:37.305028 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 219) took 3085 ms. Trace:
W20250905 19:52:37.305145 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ac5c88cf8d15433a940e937f0a225582 because the tablet was not found. No further retry: Not found: Tablet not found: ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:37.305090 2376 rpcz_store.cc:276] 0905 19:52:34.219377 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.219416 (+ 39us) service_pool.cc:225] Handling call
0905 19:52:37.304944 (+3085528us) ts_tablet_manager.cc:1145] Deleting tablet a5f076d9917f40d899d7422ad974fe0f
0905 19:52:37.304953 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.305013 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085392}
W20250905 19:52:37.305330 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 00ac56c449ba4b7b8dbd5e87120799d9 because the tablet was not found. No further retry: Not found: Tablet not found: 00ac56c449ba4b7b8dbd5e87120799d9
I20250905 19:52:37.305371 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 220) took 3085 ms. Trace:
I20250905 19:52:37.305434 2376 rpcz_store.cc:276] 0905 19:52:34.219704 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.219742 (+ 38us) service_pool.cc:225] Handling call
0905 19:52:37.305293 (+3085551us) ts_tablet_manager.cc:1145] Deleting tablet 723168b075874dfcb3eb50c5134c499e
0905 19:52:37.305302 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.305358 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085421}
W20250905 19:52:37.305512 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 03a12cf719f34096bb82f5846bbe3b2d because the tablet was not found. No further retry: Not found: Tablet not found: 03a12cf719f34096bb82f5846bbe3b2d
W20250905 19:52:37.305706 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet b13d282e1dc54d1faa2882c9b39d17cb because the tablet was not found. No further retry: Not found: Tablet not found: b13d282e1dc54d1faa2882c9b39d17cb
I20250905 19:52:37.305712 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 221) took 3085 ms. Trace:
I20250905 19:52:37.305780 2376 rpcz_store.cc:276] 0905 19:52:34.220068 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.220110 (+ 42us) service_pool.cc:225] Handling call
0905 19:52:37.305644 (+3085534us) ts_tablet_manager.cc:1145] Deleting tablet a3805127dd78489e8545da7414ead904
0905 19:52:37.305651 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.305697 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085413}
W20250905 19:52:37.305903 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 302053e1611b4e22a9a2e7b60ae68f27 because the tablet was not found. No further retry: Not found: Tablet not found: 302053e1611b4e22a9a2e7b60ae68f27
I20250905 19:52:37.306066 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 222) took 3085 ms. Trace:
W20250905 19:52:37.306099 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 5b0d4be788f841c2860f5dd65275b623 because the tablet was not found. No further retry: Not found: Tablet not found: 5b0d4be788f841c2860f5dd65275b623
I20250905 19:52:37.306149 2376 rpcz_store.cc:276] 0905 19:52:34.220341 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.220424 (+ 83us) service_pool.cc:225] Handling call
0905 19:52:37.305983 (+3085559us) ts_tablet_manager.cc:1145] Deleting tablet 9e9994863c374354838aa0f817d0057e
0905 19:52:37.305990 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.306051 (+ 61us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085431}
W20250905 19:52:37.306269 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 9630f1feb6d74472b5f6f4b54151311c because the tablet was not found. No further retry: Not found: Tablet not found: 9630f1feb6d74472b5f6f4b54151311c
I20250905 19:52:37.306457 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 223) took 3085 ms. Trace:
W20250905 19:52:37.306470 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 because the tablet was not found. No further retry: Not found: Tablet not found: 8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:37.306536 2376 rpcz_store.cc:276] 0905 19:52:34.221094 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.221138 (+ 44us) service_pool.cc:225] Handling call
0905 19:52:37.306364 (+3085226us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:37.306372 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.306440 (+ 68us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085086}
W20250905 19:52:37.306658 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a5f076d9917f40d899d7422ad974fe0f because the tablet was not found. No further retry: Not found: Tablet not found: a5f076d9917f40d899d7422ad974fe0f
I20250905 19:52:37.306819 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 224) took 3085 ms. Trace:
W20250905 19:52:37.306862 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 723168b075874dfcb3eb50c5134c499e because the tablet was not found. No further retry: Not found: Tablet not found: 723168b075874dfcb3eb50c5134c499e
I20250905 19:52:37.306902 2376 rpcz_store.cc:276] 0905 19:52:34.221454 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.221495 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:37.306745 (+3085250us) ts_tablet_manager.cc:1145] Deleting tablet 3de936d2309a411eabbd59f7b73ce7f3
0905 19:52:37.306753 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.306805 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085120}
W20250905 19:52:37.307050 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet a3805127dd78489e8545da7414ead904 because the tablet was not found. No further retry: Not found: Tablet not found: a3805127dd78489e8545da7414ead904
I20250905 19:52:37.307171 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 225) took 3085 ms. Trace:
I20250905 19:52:37.307229 2376 rpcz_store.cc:276] 0905 19:52:34.221790 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.221830 (+ 40us) service_pool.cc:225] Handling call
0905 19:52:37.307102 (+3085272us) ts_tablet_manager.cc:1145] Deleting tablet 091ef981d6b54756830927ea111e78aa
0905 19:52:37.307111 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.307159 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085143}
I20250905 19:52:37.307531 2376 tablet_replica.cc:331] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:37.307762 2376 raft_consensus.cc:2241] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
W20250905 19:52:37.307845 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 9e9994863c374354838aa0f817d0057e because the tablet was not found. No further retry: Not found: Tablet not found: 9e9994863c374354838aa0f817d0057e
W20250905 19:52:37.308370 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 54628965b82e470d811eff95ffc56c5f because the tablet was not found. No further retry: Not found: Tablet not found: 54628965b82e470d811eff95ffc56c5f
W20250905 19:52:37.308868 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 3de936d2309a411eabbd59f7b73ce7f3 because the tablet was not found. No further retry: Not found: Tablet not found: 3de936d2309a411eabbd59f7b73ce7f3
I20250905 19:52:37.309170 2413 raft_consensus.cc:491] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.309448 2413 raft_consensus.cc:513] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.310218 2413 leader_election.cc:290] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.311102 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:37.312345 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:37.312584 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:37.313355 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:37.314203 1547 catalog_manager.cc:5582] T 3b53259c71d5443ebb12827ee78038fe P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:37.314728 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:37.314819 2903 raft_consensus.cc:491] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.314934 2903 raft_consensus.cc:513] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.315001 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:37.315321 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:37.315516 2903 leader_election.cc:290] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.317173 2903 raft_consensus.cc:491] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.317286 2903 raft_consensus.cc:513] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.317736 2903 leader_election.cc:290] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.318231 2652 ts_tablet_manager.cc:1918] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.318452 2652 log.cc:1199] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/54628965b82e470d811eff95ffc56c5f
I20250905 19:52:37.319022 2652 ts_tablet_manager.cc:1939] T 54628965b82e470d811eff95ffc56c5f P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:37.319074 848 leader_election.cc:343] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.319181 848 leader_election.cc:304] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.319361 2903 raft_consensus.cc:2747] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.319650 848 leader_election.cc:343] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.319860 848 leader_election.cc:343] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.320130 848 leader_election.cc:343] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.320541 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 89) took 5588 ms. Trace:
I20250905 19:52:37.320720 2652 rpcz_store.cc:276] 0905 19:52:31.732073 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.732897 (+ 824us) service_pool.cc:225] Handling call
0905 19:52:37.280752 (+5547855us) ts_tablet_manager.cc:1145] Deleting tablet 54628965b82e470d811eff95ffc56c5f
0905 19:52:37.280761 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.314688 (+ 33927us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.318049 (+ 3361us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.320498 (+ 2449us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5503020}
I20250905 19:52:37.321231 2652 tablet_replica.cc:331] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.321724 2652 raft_consensus.cc:2241] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.321978 2652 raft_consensus.cc:2270] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.322467 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32fa16972f2a42d39b338b833c185b28" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:37.322777 1045 raft_consensus.cc:2391] T 32fa16972f2a42d39b338b833c185b28 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 835daf30e4324782848290be181ae78f in current term 1: Already voted for candidate f21c498a0d964e8fbebfd1b3cb1916fe in this term.
I20250905 19:52:37.323252 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "923ad8e40a07409e8f6a0152a0b2ef75" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.324362 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbd471695e748c9831f9b0f6795aea7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.324715 1170 tablet_bootstrap.cc:492] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:37.324940 1170 ts_tablet_manager.cc:1397] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.281s user 0.007s sys 0.022s
I20250905 19:52:37.325736 1170 raft_consensus.cc:357] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.325939 1170 raft_consensus.cc:383] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.326041 1170 raft_consensus.cc:738] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.326284 1170 consensus_queue.cc:260] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:37.326877 984 consensus_peers.cc:489] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:37.329602 1170 ts_tablet_manager.cc:1428] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:37.329931 1170 tablet_bootstrap.cc:492] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:37.336831 2652 ts_tablet_manager.cc:1905] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.340459 2376 raft_consensus.cc:2270] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.341059 2376 ts_tablet_manager.cc:1905] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.342965 2652 ts_tablet_manager.cc:1918] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.343181 2652 log.cc:1199] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6bb79e3126074736a23b128a8e14fab6
I20250905 19:52:37.343627 2652 ts_tablet_manager.cc:1939] T 6bb79e3126074736a23b128a8e14fab6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.344044 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 93) took 5513 ms. Trace:
I20250905 19:52:37.344198 2652 rpcz_store.cc:276] 0905 19:52:31.830942 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.831006 (+ 64us) service_pool.cc:225] Handling call
0905 19:52:37.321147 (+5490141us) ts_tablet_manager.cc:1145] Deleting tablet 6bb79e3126074736a23b128a8e14fab6
0905 19:52:37.321161 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.339933 (+ 18772us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.342860 (+ 2927us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.343957 (+ 1097us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5489894}
I20250905 19:52:37.344691 2652 tablet_replica.cc:331] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
W20250905 19:52:37.344882 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 62773.8us
I20250905 19:52:37.344952 2652 raft_consensus.cc:2241] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:37.345100 849 leader_election.cc:304] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.345474 2652 pending_rounds.cc:70] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:37.345589 2652 pending_rounds.cc:77] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089609611124736 op_type: NO_OP noop_request { }
I20250905 19:52:37.345775 2652 raft_consensus.cc:2887] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:37.345939 2652 raft_consensus.cc:2270] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.347748 2903 raft_consensus.cc:2802] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.347870 2903 raft_consensus.cc:695] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:37.348116 2903 consensus_queue.cc:237] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.348579 2652 ts_tablet_manager.cc:1905] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.348719 1171 tablet_bootstrap.cc:492] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.348848 1171 ts_tablet_manager.cc:1397] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.164s user 0.008s sys 0.007s
I20250905 19:52:37.349769 1171 raft_consensus.cc:357] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.349982 1171 raft_consensus.cc:383] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.350054 1171 raft_consensus.cc:738] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.350286 1171 consensus_queue.cc:260] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.350764 1171 ts_tablet_manager.cc:1428] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:37.351058 1171 tablet_bootstrap.cc:492] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.352265 2903 consensus_queue.cc:1035] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.353668 1169 tablet_bootstrap.cc:492] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.353814 1169 ts_tablet_manager.cc:1397] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.354s user 0.009s sys 0.021s
I20250905 19:52:37.354558 1169 raft_consensus.cc:357] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.354743 1169 raft_consensus.cc:383] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.354839 1169 raft_consensus.cc:738] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.355052 1169 consensus_queue.cc:260] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.355566 1169 ts_tablet_manager.cc:1428] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:37.355844 1169 tablet_bootstrap.cc:492] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.358223 1169 tablet_bootstrap.cc:654] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.360669 2652 ts_tablet_manager.cc:1918] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
W20250905 19:52:37.366623 848 leader_election.cc:343] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.366935 2652 log.cc:1199] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/421b28d047124222ae9cbc603ec22cee
W20250905 19:52:37.366961 848 leader_election.cc:343] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.367173 848 leader_election.cc:343] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.367372 848 leader_election.cc:343] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.367444 848 leader_election.cc:304] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.367586 2652 ts_tablet_manager.cc:1939] T 421b28d047124222ae9cbc603ec22cee P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.368065 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 54628965b82e470d811eff95ffc56c5f (table default.Table_c815e30da2544766834b52eaefab63c4 [id=0b36fe8f4d004271aa9bbe78a44dc583]) successfully deleted
W20250905 19:52:37.368270 1479 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerAdminService.DeleteTablet -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 46936.3us
I20250905 19:52:37.368542 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6bb79e3126074736a23b128a8e14fab6 (table default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]) successfully deleted
I20250905 19:52:37.373147 1171 tablet_bootstrap.cc:654] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.373356 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 94) took 5531 ms. Trace:
I20250905 19:52:37.373471 2652 rpcz_store.cc:276] 0905 19:52:31.842310 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.842366 (+ 56us) service_pool.cc:225] Handling call
0905 19:52:37.344592 (+5502226us) ts_tablet_manager.cc:1145] Deleting tablet 421b28d047124222ae9cbc603ec22cee
0905 19:52:37.344602 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.353370 (+ 8768us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.360553 (+ 7183us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.373283 (+ 12730us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5502011}
I20250905 19:52:37.373950 2652 tablet_replica.cc:331] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.373950 1170 tablet_bootstrap.cc:654] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.373970 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 421b28d047124222ae9cbc603ec22cee (table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]) successfully deleted
I20250905 19:52:37.374212 2652 raft_consensus.cc:2241] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:37.374619 2652 raft_consensus.cc:2270] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.368041 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:37.375298 2652 ts_tablet_manager.cc:1905] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.375651 1547 catalog_manager.cc:5582] T 98e4e6d274574c0580315dcc2323c3d6 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
W20250905 19:52:37.376737 849 leader_election.cc:343] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.377092 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:37.377403 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:37.377699 2738 raft_consensus.cc:2747] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.377759 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:37.378055 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:37.379225 1547 catalog_manager.cc:5582] T cd5312852fe345d98039aeea241b7fe8 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
W20250905 19:52:37.381110 986 consensus_peers.cc:489] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6ebab2e3698f4f60a770bde9cc019600. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:37.381594 848 leader_election.cc:343] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.381716 848 leader_election.cc:304] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.381906 2738 raft_consensus.cc:2747] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.393926 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "923ad8e40a07409e8f6a0152a0b2ef75" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.395418 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbd471695e748c9831f9b0f6795aea7" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.396490 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c69a41cee5ff4f8f9bccaa513c096118" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.397946 2652 ts_tablet_manager.cc:1918] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.398092 2652 log.cc:1199] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/3c824f5591044c4198a51a25661f71e5
I20250905 19:52:37.398643 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aff2acc3f8245e1bca41f971f8e74ce" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.399505 767 raft_consensus.cc:3058] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.400735 2652 ts_tablet_manager.cc:1939] T 3c824f5591044c4198a51a25661f71e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.400913 849 leader_election.cc:304] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.401619 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 96) took 5555 ms. Trace:
I20250905 19:52:37.401902 2738 raft_consensus.cc:2747] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.403147 849 leader_election.cc:343] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.407994 2738 raft_consensus.cc:491] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.408885 2738 raft_consensus.cc:513] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.409687 2652 rpcz_store.cc:276] 0905 19:52:31.846329 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.846373 (+ 44us) service_pool.cc:225] Handling call
0905 19:52:37.373745 (+5527372us) ts_tablet_manager.cc:1145] Deleting tablet 3c824f5591044c4198a51a25661f71e5
0905 19:52:37.373866 (+ 121us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.393642 (+ 19776us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.397805 (+ 4163us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.401585 (+ 3780us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5527165}
I20250905 19:52:37.409953 2738 leader_election.cc:290] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.410233 2652 tablet_replica.cc:331] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.410418 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 3c824f5591044c4198a51a25661f71e5 (table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]) successfully deleted
I20250905 19:52:37.410532 2652 raft_consensus.cc:2241] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.410794 2652 pending_rounds.cc:70] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
W20250905 19:52:37.410876 849 leader_election.cc:343] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.410916 2652 pending_rounds.cc:77] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089608224530432 op_type: NO_OP noop_request { }
I20250905 19:52:37.411022 2652 raft_consensus.cc:2887] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:37.411186 2652 raft_consensus.cc:2270] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.412498 2652 ts_tablet_manager.cc:1905] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.414247 1169 tablet_bootstrap.cc:492] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.414444 1169 ts_tablet_manager.cc:1397] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.059s user 0.003s sys 0.000s
I20250905 19:52:37.414486 2496 consensus_queue.cc:1035] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.416150 1171 tablet_bootstrap.cc:492] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.416301 1171 ts_tablet_manager.cc:1397] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.065s user 0.000s sys 0.003s
I20250905 19:52:37.418246 3188 tablet.cc:1722] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:37.422381 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e63b27ff4e434d76a8280f90c22f4082" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.430473 1043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c69a41cee5ff4f8f9bccaa513c096118" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.431192 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aff2acc3f8245e1bca41f971f8e74ce" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.431532 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e63b27ff4e434d76a8280f90c22f4082" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.432080 1169 raft_consensus.cc:357] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.432238 1169 raft_consensus.cc:383] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.432293 1169 raft_consensus.cc:738] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.432473 1169 consensus_queue.cc:260] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.432945 1169 ts_tablet_manager.cc:1428] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:37.434515 712 leader_election.cc:304] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.436371 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65368306f86845008d4d486023e1fc29" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.420055 1171 raft_consensus.cc:357] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.424538 2789 raft_consensus.cc:491] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.438062 1171 raft_consensus.cc:383] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.438141 1171 raft_consensus.cc:738] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.438123 2789 raft_consensus.cc:513] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.438439 1171 consensus_queue.cc:260] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.438704 2789 leader_election.cc:290] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.438941 1171 ts_tablet_manager.cc:1428] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20250905 19:52:37.439231 1171 tablet_bootstrap.cc:492] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:37.425655 848 leader_election.cc:343] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.439494 848 leader_election.cc:304] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:37.439900 848 leader_election.cc:343] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.440270 848 leader_election.cc:304] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:37.440797 848 leader_election.cc:343] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.441824 1171 tablet_bootstrap.cc:654] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.440140 2903 raft_consensus.cc:2747] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.440990 2838 raft_consensus.cc:2747] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.442624 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65368306f86845008d4d486023e1fc29" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:37.443238 848 leader_election.cc:343] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.444357 771 raft_consensus.cc:3058] T 2568b9f6f37b48efbe92cef4c19d5dfb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:37.445269 848 leader_election.cc:343] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.447407 1169 tablet_bootstrap.cc:492] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.449020 1169 tablet_bootstrap.cc:654] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.451036 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:37.454409 848 leader_election.cc:343] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.454993 2738 consensus_queue.cc:1035] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:37.456326 849 leader_election.cc:343] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.457512 849 leader_election.cc:304] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.462198 2738 raft_consensus.cc:2747] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.463111 1170 tablet_bootstrap.cc:492] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:37.463295 1170 ts_tablet_manager.cc:1397] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.133s user 0.003s sys 0.003s
I20250905 19:52:37.463343 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.464509 2376 ts_tablet_manager.cc:1918] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.464625 2376 log.cc:1199] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6bb79e3126074736a23b128a8e14fab6
W20250905 19:52:37.464552 849 leader_election.cc:343] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.466354 849 leader_election.cc:304] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.467206 2496 consensus_queue.cc:1035] T 5302b79208494e48a5f2778c5f52ad01 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.468892 2738 raft_consensus.cc:2747] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.469348 849 leader_election.cc:343] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.470496 1170 raft_consensus.cc:357] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.470600 849 leader_election.cc:304] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.470758 1170 raft_consensus.cc:383] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.470839 1170 raft_consensus.cc:738] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
W20250905 19:52:37.470352 848 leader_election.cc:343] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.471120 1170 consensus_queue.cc:260] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.471750 1170 ts_tablet_manager.cc:1428] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20250905 19:52:37.472132 1170 tablet_bootstrap.cc:492] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:37.472419 2789 raft_consensus.cc:2747] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.473203 3152 raft_consensus.cc:491] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.473414 2838 raft_consensus.cc:491] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.473348 3152 raft_consensus.cc:513] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.473518 2838 raft_consensus.cc:513] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.473996 3152 leader_election.cc:290] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.474071 1170 tablet_bootstrap.cc:654] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.474256 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca646f7c162e41469f283657e72823b4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.474778 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca646f7c162e41469f283657e72823b4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:37.474810 984 leader_election.cc:343] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.474962 906 raft_consensus.cc:2466] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:37.475171 2838 leader_election.cc:290] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.475589 986 leader_election.cc:304] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.476032 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.476265 3152 raft_consensus.cc:2802] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.476356 3152 raft_consensus.cc:491] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.476426 3152 raft_consensus.cc:3058] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:37.476482 848 leader_election.cc:343] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.479161 3194 tablet.cc:1722] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:37.480154 3152 raft_consensus.cc:513] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.481019 3152 leader_election.cc:290] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.481267 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca646f7c162e41469f283657e72823b4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:37.481737 984 leader_election.cc:343] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.482247 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca646f7c162e41469f283657e72823b4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:37.482416 906 raft_consensus.cc:3058] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:37.484650 849 leader_election.cc:343] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.484757 849 leader_election.cc:304] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.484920 906 raft_consensus.cc:2466] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:37.485073 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:37.485126 2789 raft_consensus.cc:2747] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.485512 986 leader_election.cc:304] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.465426 2376 ts_tablet_manager.cc:1939] T 6bb79e3126074736a23b128a8e14fab6 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:37.486403 2652 ts_tablet_manager.cc:1918] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.486662 3188 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:59592 (request call id 268) took 1164 ms. Trace:
W20250905 19:52:37.487066 849 leader_election.cc:343] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.486814 3188 rpcz_store.cc:276] 0905 19:52:36.322248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.322541 (+ 293us) service_pool.cc:225] Handling call
0905 19:52:37.486643 (+1164102us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.393433 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:36.393556 (+ 123us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:36.393564 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:36.393571 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:36.393604 (+ 33us) alter_schema_op.cc:128] START. Timestamp: 1757101956393212
0905 19:52:36.393607 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.393793 (+ 186us) log.cc:844] Serialized 173 byte log entry
0905 19:52:37.375298 (+981505us) op_driver.cc:464] REPLICATION: finished
0905 19:52:37.418206 (+ 42908us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:37.431693 (+ 13487us) log.cc:844] Serialized 5 byte log entry
0905 19:52:37.481026 (+ 49333us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.481371 (+ 345us) log.cc:844] Serialized 14 byte log entry
0905 19:52:37.486583 (+ 5212us) alter_schema_op.cc:73] Released schema lock
0905 19:52:37.486592 (+ 9us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":42832,"prepare.queue_time_us":70526,"prepare.run_cpu_time_us":1162,"prepare.run_wall_time_us":58858,"raft.queue_time_us":255,"raft.run_cpu_time_us":341,"raft.run_wall_time_us":340,"replication_time_us":981652,"thread_start_us":178494,"threads_started":3,"wal-append.queue_time_us":56457}]]}
I20250905 19:52:37.487792 849 leader_election.cc:304] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.489022 2789 raft_consensus.cc:2747] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.491984 2789 raft_consensus.cc:491] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.486824 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 226) took 3264 ms. Trace:
I20250905 19:52:37.492877 2789 raft_consensus.cc:513] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.492801 2376 rpcz_store.cc:276] 0905 19:52:34.222129 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.222292 (+ 163us) service_pool.cc:225] Handling call
0905 19:52:37.307482 (+3085190us) ts_tablet_manager.cc:1145] Deleting tablet 6bb79e3126074736a23b128a8e14fab6
0905 19:52:37.307490 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.397094 (+ 89604us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.464378 (+ 67284us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.486806 (+ 22428us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3085038}
I20250905 19:52:37.493572 1170 tablet_bootstrap.cc:492] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:37.493587 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 229) took 3270 ms. Trace:
I20250905 19:52:37.493734 1170 ts_tablet_manager.cc:1397] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.000s
I20250905 19:52:37.493659 2376 rpcz_store.cc:276] 0905 19:52:34.223257 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.223293 (+ 36us) service_pool.cc:225] Handling call
0905 19:52:37.493466 (+3270173us) ts_tablet_manager.cc:1145] Deleting tablet 2350bd08f977424ba94ee8fa01f66efa
0905 19:52:37.493490 (+ 24us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.493570 (+ 80us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3269878}
I20250905 19:52:37.493956 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 209) took 3292 ms. Trace:
I20250905 19:52:37.494026 2376 rpcz_store.cc:276] 0905 19:52:34.201024 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.205496 (+ 4472us) service_pool.cc:225] Handling call
0905 19:52:37.493877 (+3288381us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:37.493886 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.493942 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3105551}
I20250905 19:52:37.494282 2376 tablet_replica.cc:331] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:37.494549 2789 leader_election.cc:290] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.494611 1170 raft_consensus.cc:357] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.494882 1170 raft_consensus.cc:383] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.494999 1170 raft_consensus.cc:738] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.495251 1170 consensus_queue.cc:260] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.495517 1849 raft_consensus.cc:2747] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.495824 1170 ts_tablet_manager.cc:1428] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:37.497084 2652 log.cc:1199] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/3609de83732d4ad8857372acfbb5c32a
I20250905 19:52:37.497535 2652 ts_tablet_manager.cc:1939] T 3609de83732d4ad8857372acfbb5c32a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.488404 2496 raft_consensus.cc:2802] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.498281 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 95) took 5652 ms. Trace:
I20250905 19:52:37.498322 2496 raft_consensus.cc:695] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:37.498359 2652 rpcz_store.cc:276] 0905 19:52:31.846213 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:31.846651 (+ 438us) service_pool.cc:225] Handling call
0905 19:52:37.410144 (+5563493us) ts_tablet_manager.cc:1145] Deleting tablet 3609de83732d4ad8857372acfbb5c32a
0905 19:52:37.410158 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.483410 (+ 73252us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.486300 (+ 2890us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.498258 (+ 11958us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":5563355}
I20250905 19:52:37.498579 2496 consensus_queue.cc:237] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.498703 2652 tablet_replica.cc:331] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.498904 2652 raft_consensus.cc:2241] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:37.499217 2652 pending_rounds.cc:70] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:37.499264 2652 pending_rounds.cc:77] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089612439691264 op_type: NO_OP noop_request { }
I20250905 19:52:37.499361 2652 raft_consensus.cc:2887] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:37.499449 2652 raft_consensus.cc:2270] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.500129 2652 ts_tablet_manager.cc:1905] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.510560 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6bb79e3126074736a23b128a8e14fab6 (table default.Table_9c5296ef0f424c5591eacb019ed29c7d [id=428812067f01437eb45d074d6d8eabee]) successfully deleted
I20250905 19:52:37.494554 2376 raft_consensus.cc:2241] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:37.511637 2376 raft_consensus.cc:2270] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.514976 2376 ts_tablet_manager.cc:1905] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.520001 2652 ts_tablet_manager.cc:1918] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.524065 2652 log.cc:1199] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6ebab2e3698f4f60a770bde9cc019600
I20250905 19:52:37.524549 2652 ts_tablet_manager.cc:1939] T 6ebab2e3698f4f60a770bde9cc019600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.526331 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 97) took 4783 ms. Trace:
I20250905 19:52:37.526475 2652 rpcz_store.cc:276] 0905 19:52:32.743126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.743184 (+ 58us) service_pool.cc:225] Handling call
0905 19:52:37.498646 (+4755462us) ts_tablet_manager.cc:1145] Deleting tablet 6ebab2e3698f4f60a770bde9cc019600
0905 19:52:37.498658 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.516686 (+ 18028us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.519861 (+ 3175us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.526312 (+ 6451us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4755247}
I20250905 19:52:37.526937 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 98) took 4783 ms. Trace:
I20250905 19:52:37.527058 2652 rpcz_store.cc:276] 0905 19:52:32.743950 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.744028 (+ 78us) service_pool.cc:225] Handling call
0905 19:52:37.526841 (+4782813us) ts_tablet_manager.cc:1145] Deleting tablet 578a6b9224cc4d8585d6c4b241762e1b
0905 19:52:37.526850 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.526923 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4782637}
I20250905 19:52:37.527424 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 101) took 4776 ms. Trace:
I20250905 19:52:37.527541 2652 rpcz_store.cc:276] 0905 19:52:32.750799 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.750850 (+ 51us) service_pool.cc:225] Handling call
0905 19:52:37.527345 (+4776495us) ts_tablet_manager.cc:1145] Deleting tablet fd793bce71e344f18bb177a53ab1bfc1
0905 19:52:37.527354 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.527411 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4776281}
W20250905 19:52:37.529711 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 2350bd08f977424ba94ee8fa01f66efa because the tablet was not found. No further retry: Not found: Tablet not found: 2350bd08f977424ba94ee8fa01f66efa
I20250905 19:52:37.529978 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:37.530030 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff because the tablet was not found. No further retry: Not found: Tablet not found: c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:37.530213 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 3609de83732d4ad8857372acfbb5c32a (table default.Table_f6e643730c10465cb8585cb48c1c5e04 [id=fa983711b6574ebeb9dfb2cca302bdc1]) successfully deleted
I20250905 19:52:37.530483 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.530778 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6ebab2e3698f4f60a770bde9cc019600 (table default.Table_68450db54c5641a1a249cdcf5fa2d5ec [id=94f2531a27494c32bcd549a25287867f]) successfully deleted
W20250905 19:52:37.531008 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 578a6b9224cc4d8585d6c4b241762e1b because the tablet was not found. No further retry: Not found: Tablet not found: 578a6b9224cc4d8585d6c4b241762e1b
W20250905 19:52:37.531203 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet fd793bce71e344f18bb177a53ab1bfc1 because the tablet was not found. No further retry: Not found: Tablet not found: fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:37.532110 1547 catalog_manager.cc:5582] T a311f9b943d143a9a52d7771ce7186d7 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:37.532306 2738 raft_consensus.cc:491] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.532590 2738 raft_consensus.cc:513] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.533169 2738 leader_election.cc:290] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.533169 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 102) took 4780 ms. Trace:
I20250905 19:52:37.533257 2652 rpcz_store.cc:276] 0905 19:52:32.752216 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.752263 (+ 47us) service_pool.cc:225] Handling call
0905 19:52:37.527822 (+4775559us) ts_tablet_manager.cc:1145] Deleting tablet d296428b70f9411190b6c408d81639f2
0905 19:52:37.529891 (+ 2069us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.533153 (+ 3262us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4775380}
I20250905 19:52:37.533632 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 103) took 4780 ms. Trace:
I20250905 19:52:37.533739 2652 rpcz_store.cc:276] 0905 19:52:32.753618 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.753665 (+ 47us) service_pool.cc:225] Handling call
0905 19:52:37.533529 (+4779864us) ts_tablet_manager.cc:1145] Deleting tablet 2350bd08f977424ba94ee8fa01f66efa
0905 19:52:37.533539 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.533615 (+ 76us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4779690}
I20250905 19:52:37.534152 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 104) took 4778 ms. Trace:
W20250905 19:52:37.536284 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet d296428b70f9411190b6c408d81639f2 because the tablet was not found. No further retry: Not found: Tablet not found: d296428b70f9411190b6c408d81639f2
I20250905 19:52:37.536437 2652 rpcz_store.cc:276] 0905 19:52:32.755210 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.755278 (+ 68us) service_pool.cc:225] Handling call
0905 19:52:37.533970 (+4778692us) ts_tablet_manager.cc:1145] Deleting tablet f48f9f789b6e46ec871a0fd007770dfc
0905 19:52:37.534070 (+ 100us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.534135 (+ 65us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4778541}
W20250905 19:52:37.536502 849 leader_election.cc:343] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.536634 849 leader_election.cc:304] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.536849 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 105) took 4781 ms. Trace:
I20250905 19:52:37.536978 2652 rpcz_store.cc:276] 0905 19:52:32.755501 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.755558 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:37.536766 (+4781208us) ts_tablet_manager.cc:1145] Deleting tablet 24bcd21f19ba465aa550102e0fccb674
0905 19:52:37.536776 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.536833 (+ 57us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4781007}
W20250905 19:52:37.537298 849 leader_election.cc:343] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.537361 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 107) took 4779 ms. Trace:
I20250905 19:52:37.537532 2652 rpcz_store.cc:276] 0905 19:52:32.757853 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.757956 (+ 103us) service_pool.cc:225] Handling call
0905 19:52:37.537277 (+4779321us) ts_tablet_manager.cc:1145] Deleting tablet 37894e39e6f2417fbfcd6d43f1ea2b37
0905 19:52:37.537286 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.537346 (+ 60us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4777261}
I20250905 19:52:37.538182 2903 raft_consensus.cc:2747] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.538347 1171 tablet_bootstrap.cc:492] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.538498 1171 ts_tablet_manager.cc:1397] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.099s user 0.004s sys 0.004s
I20250905 19:52:37.539220 1171 raft_consensus.cc:357] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.539374 1171 raft_consensus.cc:383] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.539417 1171 raft_consensus.cc:738] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.539587 1171 consensus_queue.cc:260] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.539780 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74eb2632a6414e52b9ad67f53c182679" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.540064 1171 ts_tablet_manager.cc:1428] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 19:52:37.540320 1171 tablet_bootstrap.cc:492] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.540894 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 108) took 4782 ms. Trace:
I20250905 19:52:37.541018 2652 rpcz_store.cc:276] 0905 19:52:32.758340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.758384 (+ 44us) service_pool.cc:225] Handling call
0905 19:52:37.537814 (+4779430us) ts_tablet_manager.cc:1145] Deleting tablet a2ff206366ad4344b4df5fc6a5584f8a
0905 19:52:37.538371 (+ 557us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.540875 (+ 2504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4777740}
I20250905 19:52:37.541501 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 109) took 4782 ms. Trace:
I20250905 19:52:37.541625 2652 rpcz_store.cc:276] 0905 19:52:32.758752 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.758798 (+ 46us) service_pool.cc:225] Handling call
0905 19:52:37.541420 (+4782622us) ts_tablet_manager.cc:1145] Deleting tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e
0905 19:52:37.541429 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.541485 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4781304}
I20250905 19:52:37.541982 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 110) took 4782 ms. Trace:
I20250905 19:52:37.542102 2652 rpcz_store.cc:276] 0905 19:52:32.759112 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.759155 (+ 43us) service_pool.cc:225] Handling call
0905 19:52:37.541906 (+4782751us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:37.541915 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.541969 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4781753}
I20250905 19:52:37.543907 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 106) took 4786 ms. Trace:
I20250905 19:52:37.544064 2652 rpcz_store.cc:276] 0905 19:52:32.757576 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:32.759283 (+ 1707us) service_pool.cc:225] Handling call
0905 19:52:37.542392 (+4783109us) ts_tablet_manager.cc:1145] Deleting tablet a1b4989f17454aa58c05727b797c2009
0905 19:52:37.542400 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.543891 (+ 1491us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":4782169}
W20250905 19:52:37.544637 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 2350bd08f977424ba94ee8fa01f66efa because the tablet was not found. No further retry: Not found: Tablet not found: 2350bd08f977424ba94ee8fa01f66efa
W20250905 19:52:37.545017 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet f48f9f789b6e46ec871a0fd007770dfc because the tablet was not found. No further retry: Not found: Tablet not found: f48f9f789b6e46ec871a0fd007770dfc
W20250905 19:52:37.545253 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 24bcd21f19ba465aa550102e0fccb674 because the tablet was not found. No further retry: Not found: Tablet not found: 24bcd21f19ba465aa550102e0fccb674
W20250905 19:52:37.545480 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 37894e39e6f2417fbfcd6d43f1ea2b37 because the tablet was not found. No further retry: Not found: Tablet not found: 37894e39e6f2417fbfcd6d43f1ea2b37
I20250905 19:52:37.545293 2652 tablet_replica.cc:331] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
W20250905 19:52:37.545679 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a2ff206366ad4344b4df5fc6a5584f8a because the tablet was not found. No further retry: Not found: Tablet not found: a2ff206366ad4344b4df5fc6a5584f8a
W20250905 19:52:37.545862 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet cbfd03ad0bd34b6ca86e7b07a62b2a8e because the tablet was not found. No further retry: Not found: Tablet not found: cbfd03ad0bd34b6ca86e7b07a62b2a8e
W20250905 19:52:37.546074 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because the tablet was not found. No further retry: Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:37.546252 2652 raft_consensus.cc:2241] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:37.546458 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a1b4989f17454aa58c05727b797c2009 because the tablet was not found. No further retry: Not found: Tablet not found: a1b4989f17454aa58c05727b797c2009
I20250905 19:52:37.548254 2652 raft_consensus.cc:2270] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.548291 1547 catalog_manager.cc:5582] T 1d095804b51b4205a23ac3d3106312f2 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
W20250905 19:52:37.548513 849 leader_election.cc:343] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.548879 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:37.549189 1547 catalog_manager.cc:5582] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:37.549580 1547 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:37.552482 2652 ts_tablet_manager.cc:1905] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.555173 1169 tablet_bootstrap.cc:492] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.555505 1169 ts_tablet_manager.cc:1397] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.108s user 0.006s sys 0.000s
I20250905 19:52:37.556371 1169 raft_consensus.cc:357] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.556723 1169 raft_consensus.cc:383] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.556835 1169 raft_consensus.cc:738] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.557099 1169 consensus_queue.cc:260] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.557706 1169 ts_tablet_manager.cc:1428] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:37.562865 1171 tablet_bootstrap.cc:654] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.565408 1169 tablet_bootstrap.cc:492] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.566843 1169 tablet_bootstrap.cc:654] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.571720 2652 ts_tablet_manager.cc:1918] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.573753 2738 raft_consensus.cc:491] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.573889 2738 raft_consensus.cc:513] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.574489 2738 leader_election.cc:290] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.576633 2652 log.cc:1199] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/c14dc571b586405f8fd1eaa452998978
I20250905 19:52:37.577255 2652 ts_tablet_manager.cc:1939] T c14dc571b586405f8fd1eaa452998978 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.579887 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.582358 1169 tablet_bootstrap.cc:492] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.582598 1169 ts_tablet_manager.cc:1397] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.002s
I20250905 19:52:37.583575 1169 raft_consensus.cc:357] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.583900 1169 raft_consensus.cc:383] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.584056 1169 raft_consensus.cc:738] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.584390 1169 consensus_queue.cc:260] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.585062 1169 ts_tablet_manager.cc:1428] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
W20250905 19:52:37.585193 849 leader_election.cc:343] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.590910 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.591218 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74eb2632a6414e52b9ad67f53c182679" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.591559 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:37.592795 848 leader_election.cc:343] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.592917 848 leader_election.cc:304] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.593122 2738 raft_consensus.cc:2747] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.593379 848 leader_election.cc:343] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.593475 848 leader_election.cc:304] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.593641 2738 raft_consensus.cc:2747] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.593786 848 leader_election.cc:343] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.593871 848 leader_election.cc:304] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.594031 2738 raft_consensus.cc:2747] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.594445 2738 consensus_queue.cc:1035] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.603847 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 113) took 3595 ms. Trace:
I20250905 19:52:37.604300 2652 rpcz_store.cc:276] 0905 19:52:34.008302 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.008536 (+ 234us) service_pool.cc:225] Handling call
0905 19:52:37.545246 (+3536710us) ts_tablet_manager.cc:1145] Deleting tablet c14dc571b586405f8fd1eaa452998978
0905 19:52:37.545256 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.561892 (+ 16636us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.571616 (+ 9724us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.603822 (+ 32206us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3536470}
I20250905 19:52:37.605993 2652 tablet_replica.cc:331] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.606138 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet c14dc571b586405f8fd1eaa452998978 (table default.Table_2d5b20700ad74baf8138bdcadb0bef37 [id=f3e1ab5954464d628ef350897a220be0]) successfully deleted
I20250905 19:52:37.606928 2652 raft_consensus.cc:2241] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:37.606575 1169 tablet_bootstrap.cc:492] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.608793 2422 raft_consensus.cc:491] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.609033 1169 tablet_bootstrap.cc:654] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.608960 2422 raft_consensus.cc:513] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.609308 2652 pending_rounds.cc:70] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:37.609356 2652 pending_rounds.cc:77] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089615852765184 op_type: NO_OP noop_request { }
I20250905 19:52:37.609432 2652 raft_consensus.cc:2887] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:37.609514 2652 raft_consensus.cc:2270] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.609567 2422 leader_election.cc:290] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.610131 2422 consensus_queue.cc:1035] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.610828 2652 ts_tablet_manager.cc:1905] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.611069 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c870e689af254fb0acb15330577e5c42" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:37.626847 849 leader_election.cc:343] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.628870 3188 tablet.cc:1722] T 040e65fdb1104b07b953044f9b6b538a P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:37.631469 2652 ts_tablet_manager.cc:1918] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.632112 1849 raft_consensus.cc:491] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.644192 2652 log.cc:1199] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/14d5eb8bec074d2c8766fe7b159031e5
I20250905 19:52:37.639173 2496 raft_consensus.cc:491] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.644609 2496 raft_consensus.cc:513] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.645272 2496 leader_election.cc:290] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.645484 1171 tablet_bootstrap.cc:492] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.645618 1171 ts_tablet_manager.cc:1397] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.105s user 0.003s sys 0.001s
I20250905 19:52:37.646377 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.646559 907 raft_consensus.cc:2466] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:37.647533 986 leader_election.cc:304] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.647884 2496 raft_consensus.cc:2802] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.647992 2496 raft_consensus.cc:491] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.648074 2496 raft_consensus.cc:3058] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.651337 2496 raft_consensus.cc:513] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.652596 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:37.652766 907 raft_consensus.cc:3058] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.658650 907 raft_consensus.cc:2466] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:37.670928 2496 leader_election.cc:290] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.671937 2789 raft_consensus.cc:491] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.672044 986 leader_election.cc:304] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.672107 2789 raft_consensus.cc:513] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.672318 2496 raft_consensus.cc:2802] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.672425 2496 raft_consensus.cc:695] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:37.672708 2789 leader_election.cc:290] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.672648 2496 consensus_queue.cc:237] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.673859 3152 raft_consensus.cc:491] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.674019 3152 raft_consensus.cc:513] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.674607 3152 leader_election.cc:290] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.641320 3194 tablet.cc:1722] T 040e65fdb1104b07b953044f9b6b538a P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:37.675626 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10b4e82c4bc429b9a42db39c03a5267" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.675915 907 raft_consensus.cc:2466] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:37.676327 986 leader_election.cc:304] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.676573 3152 raft_consensus.cc:2802] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.676647 3152 raft_consensus.cc:491] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.676731 3152 raft_consensus.cc:3058] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.679306 3152 raft_consensus.cc:513] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.684350 2789 raft_consensus.cc:491] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.684554 2789 raft_consensus.cc:513] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.685274 2789 leader_election.cc:290] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.636308 1169 tablet_bootstrap.cc:492] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.685622 1169 ts_tablet_manager.cc:1397] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.079s user 0.004s sys 0.002s
I20250905 19:52:37.685739 882 tablet_service.cc:1468] Processing CreateTablet for tablet 30e1ca2bd1f94f748f2022b5669bc07e (DEFAULT_TABLE table=default.Table_85e1b8aa7bd340459631f0b02ae94b9d [id=151cf052220f4749be3757d20864c425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:37.685896 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 30e1ca2bd1f94f748f2022b5669bc07e (DEFAULT_TABLE table=default.Table_85e1b8aa7bd340459631f0b02ae94b9d [id=151cf052220f4749be3757d20864c425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:37.686215 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e1ca2bd1f94f748f2022b5669bc07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:37.686262 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e1ca2bd1f94f748f2022b5669bc07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:37.686316 1169 raft_consensus.cc:357] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.686445 1169 raft_consensus.cc:383] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.686492 1169 raft_consensus.cc:738] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.688405 1169 consensus_queue.cc:260] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.688889 1169 ts_tablet_manager.cc:1428] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:37.689138 1169 tablet_bootstrap.cc:492] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.690428 1169 tablet_bootstrap.cc:654] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.679889 3152 leader_election.cc:290] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.698326 2789 raft_consensus.cc:491] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.698566 2789 raft_consensus.cc:513] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.699638 2789 leader_election.cc:290] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.702193 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10b4e82c4bc429b9a42db39c03a5267" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:37.702448 907 raft_consensus.cc:3058] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.703855 2652 ts_tablet_manager.cc:1939] T 14d5eb8bec074d2c8766fe7b159031e5 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.706321 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 119) took 3524 ms. Trace:
I20250905 19:52:37.706683 907 raft_consensus.cc:2466] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:37.708174 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e8bfe047a6e4cc087eea75244da5b41" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.709213 1169 tablet_bootstrap.cc:492] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.709970 1169 ts_tablet_manager.cc:1397] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.001s
I20250905 19:52:37.709725 2652 rpcz_store.cc:276] 0905 19:52:34.192450 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.192578 (+ 128us) service_pool.cc:225] Handling call
0905 19:52:37.605940 (+3413362us) ts_tablet_manager.cc:1145] Deleting tablet 14d5eb8bec074d2c8766fe7b159031e5
0905 19:52:37.605951 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.626922 (+ 20971us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.631368 (+ 4446us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.706298 (+ 74930us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3411400}
I20250905 19:52:37.710615 1169 raft_consensus.cc:357] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.710785 1169 raft_consensus.cc:383] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.710853 1169 raft_consensus.cc:738] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.711020 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 14d5eb8bec074d2c8766fe7b159031e5 (table default.Table_b21bb0be9ae1435c8929de3f6b662d8c [id=a9a1d004ba9f4415b2fb958c789d66ba]) successfully deleted
I20250905 19:52:37.711048 1169 consensus_queue.cc:260] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.711536 1169 ts_tablet_manager.cc:1428] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:37.711802 1169 tablet_bootstrap.cc:492] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.711894 986 leader_election.cc:304] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.712159 3151 raft_consensus.cc:2802] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.712260 3151 raft_consensus.cc:695] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:37.712491 3151 consensus_queue.cc:237] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.715315 2838 raft_consensus.cc:491] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.715481 2838 raft_consensus.cc:513] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.716079 2838 leader_election.cc:290] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.717643 1170 tablet_bootstrap.cc:492] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:37.717990 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c870e689af254fb0acb15330577e5c42" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.719825 1169 tablet_bootstrap.cc:654] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.720726 2652 tablet_replica.cc:331] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.721271 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.719640 1849 raft_consensus.cc:513] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.721639 2652 raft_consensus.cc:2241] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.723619 2652 raft_consensus.cc:2270] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.723621 3152 consensus_queue.cc:1035] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:52:37.724581 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.724922 751 tablet_service.cc:1468] Processing CreateTablet for tablet 30e1ca2bd1f94f748f2022b5669bc07e (DEFAULT_TABLE table=default.Table_85e1b8aa7bd340459631f0b02ae94b9d [id=151cf052220f4749be3757d20864c425]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:37.725896 2652 ts_tablet_manager.cc:1905] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.727525 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10b4e82c4bc429b9a42db39c03a5267" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.727854 767 raft_consensus.cc:3058] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.728086 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.728900 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e10b4e82c4bc429b9a42db39c03a5267" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:37.729195 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:37.733247 984 leader_election.cc:343] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.733711 984 leader_election.cc:343] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.733959 984 leader_election.cc:343] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.734232 984 leader_election.cc:343] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.737515 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30e1ca2bd1f94f748f2022b5669bc07e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:37.740974 1171 raft_consensus.cc:357] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.741191 1171 raft_consensus.cc:383] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.741254 1171 raft_consensus.cc:738] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.741530 1171 consensus_queue.cc:260] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.742100 1171 ts_tablet_manager.cc:1428] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.096s user 0.001s sys 0.001s
I20250905 19:52:37.742429 1171 tablet_bootstrap.cc:492] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.744429 1171 tablet_bootstrap.cc:654] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.744596 1170 tablet_bootstrap.cc:654] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.746052 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e8bfe047a6e4cc087eea75244da5b41" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.747846 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc5428f20ca14fdf88a3a52f7b4d1e19" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:37.748319 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc5428f20ca14fdf88a3a52f7b4d1e19" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.749046 1849 leader_election.cc:290] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.749542 766 raft_consensus.cc:3058] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:37.757745 849 leader_election.cc:343] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.759096 2376 ts_tablet_manager.cc:1918] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:37.759298 2376 log.cc:1199] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/dc95a83ca872495894ef439cdda4e364
I20250905 19:52:37.759608 2652 ts_tablet_manager.cc:1918] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.759789 2376 ts_tablet_manager.cc:1939] T dc95a83ca872495894ef439cdda4e364 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:37.760648 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 237) took 1808 ms. Trace:
I20250905 19:52:37.760756 2376 rpcz_store.cc:276] 0905 19:52:35.951907 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.952002 (+ 95us) service_pool.cc:225] Handling call
0905 19:52:37.494235 (+1542233us) ts_tablet_manager.cc:1145] Deleting tablet dc95a83ca872495894ef439cdda4e364
0905 19:52:37.494242 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.565476 (+ 71234us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.758985 (+193509us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.760628 (+ 1643us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1542045}
I20250905 19:52:37.761368 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]) successfully deleted
I20250905 19:52:37.762266 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 243) took 1809 ms. Trace:
I20250905 19:52:37.762449 2376 rpcz_store.cc:276] 0905 19:52:35.952461 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.953745 (+ 1284us) service_pool.cc:225] Handling call
0905 19:52:37.762139 (+1808394us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:37.762152 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.762245 (+ 93us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1808213}
I20250905 19:52:37.764595 2652 log.cc:1199] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9193487b4ab74dafa76a386ddcaf674b
I20250905 19:52:37.765420 2652 ts_tablet_manager.cc:1939] T 9193487b4ab74dafa76a386ddcaf674b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.781080 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:37.787324 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 5bd9380e8ffc432da4a684934c2df68f because the tablet was not found. No further retry: Not found: Tablet not found: 5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:37.787729 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 246) took 1821 ms. Trace:
I20250905 19:52:37.787817 2376 rpcz_store.cc:276] 0905 19:52:35.965849 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.965905 (+ 56us) service_pool.cc:225] Handling call
0905 19:52:37.787623 (+1821718us) ts_tablet_manager.cc:1145] Deleting tablet 6f3473785fff45d692bd50b1015ffe39
0905 19:52:37.787635 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.787710 (+ 75us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1821509}
W20250905 19:52:37.788293 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 6f3473785fff45d692bd50b1015ffe39 because the tablet was not found. No further retry: Not found: Tablet not found: 6f3473785fff45d692bd50b1015ffe39
I20250905 19:52:37.788612 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 249) took 1822 ms. Trace:
I20250905 19:52:37.788681 2376 rpcz_store.cc:276] 0905 19:52:35.966153 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.966192 (+ 39us) service_pool.cc:225] Handling call
0905 19:52:37.788536 (+1822344us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:37.788545 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.788598 (+ 53us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1822208}
W20250905 19:52:37.789029 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet c46ea498d6a14d1ebdffea415cfc7114 because the tablet was not found. No further retry: Not found: Tablet not found: c46ea498d6a14d1ebdffea415cfc7114
I20250905 19:52:37.789306 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 251) took 1822 ms. Trace:
I20250905 19:52:37.789379 2376 rpcz_store.cc:276] 0905 19:52:35.966345 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.966382 (+ 37us) service_pool.cc:225] Handling call
0905 19:52:37.789225 (+1822843us) ts_tablet_manager.cc:1145] Deleting tablet d296428b70f9411190b6c408d81639f2
0905 19:52:37.789235 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.789291 (+ 56us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1822732}
W20250905 19:52:37.789767 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet d296428b70f9411190b6c408d81639f2 because the tablet was not found. No further retry: Not found: Tablet not found: d296428b70f9411190b6c408d81639f2
W20250905 19:52:37.770172 849 leader_election.cc:343] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.785477 848 leader_election.cc:343] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.789949 848 leader_election.cc:304] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.790226 2838 raft_consensus.cc:2747] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.790649 848 leader_election.cc:343] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.790746 848 leader_election.cc:304] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.791052 2838 raft_consensus.cc:2747] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.791090 848 leader_election.cc:343] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.791180 848 leader_election.cc:304] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.791383 2838 raft_consensus.cc:2747] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.791496 848 leader_election.cc:343] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.791822 848 leader_election.cc:343] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:37.793258 849 leader_election.cc:343] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.793360 849 leader_election.cc:304] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.785454 2789 consensus_queue.cc:1035] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.794130 2838 raft_consensus.cc:2747] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.808056 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:37.810361 1170 tablet_bootstrap.cc:492] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:37.810587 1170 ts_tablet_manager.cc:1397] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.214s user 0.004s sys 0.008s
I20250905 19:52:37.813529 1170 raft_consensus.cc:357] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.814174 1170 raft_consensus.cc:383] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.814275 1170 raft_consensus.cc:738] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.814512 1170 consensus_queue.cc:260] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.781826 1169 tablet_bootstrap.cc:492] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.785823 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fa5d34f9c0ee4d55ade61b30b0f4c293" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.816897 1169 ts_tablet_manager.cc:1397] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.105s user 0.010s sys 0.006s
I20250905 19:52:37.817052 1044 raft_consensus.cc:2391] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:37.817552 1169 raft_consensus.cc:357] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.817710 1169 raft_consensus.cc:383] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.817759 1169 raft_consensus.cc:738] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.817938 1169 consensus_queue.cc:260] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.818401 1169 ts_tablet_manager.cc:1428] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:37.818674 1169 tablet_bootstrap.cc:492] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.818992 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 124) took 3611 ms. Trace:
I20250905 19:52:37.797224 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.795712 2931 consensus_queue.cc:1035] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.820802 1169 tablet_bootstrap.cc:654] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.829129 1170 ts_tablet_manager.cc:1428] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250905 19:52:37.829519 1170 tablet_bootstrap.cc:492] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:37.831355 1170 tablet_bootstrap.cc:654] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.832465 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 250) took 1866 ms. Trace:
I20250905 19:52:37.832569 2376 rpcz_store.cc:276] 0905 19:52:35.966252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.966538 (+ 286us) service_pool.cc:225] Handling call
0905 19:52:37.832346 (+1865808us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:37.832360 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.832443 (+ 83us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1865601}
W20250905 19:52:37.833047 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 8503f6918ffb44efa36214633c93e31d because the tablet was not found. No further retry: Not found: Tablet not found: 8503f6918ffb44efa36214633c93e31d
W20250905 19:52:37.833698 849 leader_election.cc:343] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.834115 849 leader_election.cc:304] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.833945 1171 tablet_bootstrap.cc:492] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.834578 1171 ts_tablet_manager.cc:1397] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.092s user 0.002s sys 0.001s
I20250905 19:52:37.836887 2422 raft_consensus.cc:2747] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.837854 712 leader_election.cc:304] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.838280 1849 raft_consensus.cc:3058] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.839286 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 252) took 1872 ms. Trace:
I20250905 19:52:37.839366 2376 rpcz_store.cc:276] 0905 19:52:35.966455 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.966782 (+ 327us) service_pool.cc:225] Handling call
0905 19:52:37.839188 (+1872406us) ts_tablet_manager.cc:1145] Deleting tablet fd793bce71e344f18bb177a53ab1bfc1
0905 19:52:37.839198 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.839269 (+ 71us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872201}
W20250905 19:52:37.839828 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet fd793bce71e344f18bb177a53ab1bfc1 because the tablet was not found. No further retry: Not found: Tablet not found: fd793bce71e344f18bb177a53ab1bfc1
I20250905 19:52:37.840910 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 248) took 1874 ms. Trace:
I20250905 19:52:37.840996 2376 rpcz_store.cc:276] 0905 19:52:35.966062 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.966976 (+ 914us) service_pool.cc:225] Handling call
0905 19:52:37.840825 (+1873849us) ts_tablet_manager.cc:1145] Deleting tablet 44769f1170354870ae2d34fdb67ffbae
0905 19:52:37.840834 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.840896 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1873738}
I20250905 19:52:37.841276 1849 raft_consensus.cc:2747] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.841454 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 44769f1170354870ae2d34fdb67ffbae because the tablet was not found. No further retry: Not found: Tablet not found: 44769f1170354870ae2d34fdb67ffbae
I20250905 19:52:37.842070 3188 tablet.cc:1722] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:37.842943 1169 tablet_bootstrap.cc:492] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.843066 1169 ts_tablet_manager.cc:1397] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250905 19:52:37.843710 1169 raft_consensus.cc:357] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.843889 1169 raft_consensus.cc:383] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.843940 1169 raft_consensus.cc:738] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.844166 1169 consensus_queue.cc:260] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.844965 1169 ts_tablet_manager.cc:1428] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:37.845248 1169 tablet_bootstrap.cc:492] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.845636 2652 rpcz_store.cc:276] 0905 19:52:34.207248 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.207752 (+ 504us) service_pool.cc:225] Handling call
0905 19:52:37.719558 (+3511806us) ts_tablet_manager.cc:1145] Deleting tablet 9193487b4ab74dafa76a386ddcaf674b
0905 19:52:37.720671 (+ 1113us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.744613 (+ 23942us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.759502 (+ 14889us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.818973 (+ 59471us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":3,"tablet-delete.queue_time_us":3511594}
I20250905 19:52:37.846112 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9193487b4ab74dafa76a386ddcaf674b (table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]) successfully deleted
I20250905 19:52:37.846482 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 125) took 3639 ms. Trace:
I20250905 19:52:37.848163 3226 tablet.cc:1722] T 040e65fdb1104b07b953044f9b6b538a P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:37.848620 1171 raft_consensus.cc:357] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.848784 1171 raft_consensus.cc:383] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.848832 1171 raft_consensus.cc:738] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.849012 1171 consensus_queue.cc:260] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.849500 1171 ts_tablet_manager.cc:1428] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.015s user 0.000s sys 0.000s
I20250905 19:52:37.849771 1171 tablet_bootstrap.cc:492] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.851114 1171 tablet_bootstrap.cc:654] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.854998 3226 tablet.cc:1722] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:37.847615 1169 tablet_bootstrap.cc:654] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.864722 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 247) took 1898 ms. Trace:
I20250905 19:52:37.864848 2376 rpcz_store.cc:276] 0905 19:52:35.965974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.967112 (+ 1138us) service_pool.cc:225] Handling call
0905 19:52:37.864595 (+1897483us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:37.864609 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.864697 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1897362}
I20250905 19:52:37.847750 3194 tablet.cc:1722] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250905 19:52:37.865386 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet ba161282480e4736889559a2e37b195f because the tablet was not found. No further retry: Not found: Tablet not found: ba161282480e4736889559a2e37b195f
I20250905 19:52:37.865433 2422 raft_consensus.cc:491] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.865561 2422 raft_consensus.cc:513] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.866099 2422 leader_election.cc:290] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.866688 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:37.867157 848 leader_election.cc:343] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.867833 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:37.868369 2652 rpcz_store.cc:276] 0905 19:52:34.207376 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.207979 (+ 603us) service_pool.cc:225] Handling call
0905 19:52:37.846382 (+3638403us) ts_tablet_manager.cc:1145] Deleting tablet 53ac46ce05694bc2b53e195638be9e23
0905 19:52:37.846404 (+ 22us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.846466 (+ 62us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3638256}
W20250905 19:52:37.868597 849 leader_election.cc:343] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.868984 849 leader_election.cc:304] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.869642 2422 raft_consensus.cc:2747] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:37.870414 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 53ac46ce05694bc2b53e195638be9e23 because the tablet was not found. No further retry: Not found: Tablet not found: 53ac46ce05694bc2b53e195638be9e23
I20250905 19:52:37.878396 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 121) took 3671 ms. Trace:
I20250905 19:52:37.878592 2652 rpcz_store.cc:276] 0905 19:52:34.206892 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.207703 (+ 811us) service_pool.cc:225] Handling call
0905 19:52:37.870689 (+3662986us) ts_tablet_manager.cc:1145] Deleting tablet 5b0d4be788f841c2860f5dd65275b623
0905 19:52:37.878265 (+ 7576us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.878359 (+ 94us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3661914}
W20250905 19:52:37.879184 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 5b0d4be788f841c2860f5dd65275b623 because the tablet was not found. No further retry: Not found: Tablet not found: 5b0d4be788f841c2860f5dd65275b623
I20250905 19:52:37.880296 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 266) took 1872 ms. Trace:
I20250905 19:52:37.880395 2376 rpcz_store.cc:276] 0905 19:52:36.007346 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.007387 (+ 41us) service_pool.cc:225] Handling call
0905 19:52:37.880189 (+1872802us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:37.880201 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.880275 (+ 74us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1872622}
W20250905 19:52:37.880836 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because the tablet was not found. No further retry: Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:37.880824 3188 tablet.cc:1722] T 3fe2b75a3e59437c885abc18614f1da2 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:37.881138 2652 tablet_replica.cc:331] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:37.881194 2376 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:59592 (request call id 267) took 1865 ms. Trace:
I20250905 19:52:37.881270 2376 rpcz_store.cc:276] 0905 19:52:36.015446 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.015503 (+ 57us) service_pool.cc:225] Handling call
0905 19:52:37.881080 (+1865577us) ts_tablet_manager.cc:1145] Deleting tablet 52c99a76802b481c98f77dbd90de7cbb
0905 19:52:37.881089 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.881177 (+ 88us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1865408}
I20250905 19:52:37.881546 2652 raft_consensus.cc:2241] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:37.881666 1484 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet 52c99a76802b481c98f77dbd90de7cbb because the tablet was not found. No further retry: Not found: Tablet not found: 52c99a76802b481c98f77dbd90de7cbb
I20250905 19:52:37.881836 2652 raft_consensus.cc:2270] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:37.882355 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:37.882578 2652 ts_tablet_manager.cc:1905] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.882629 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:37.884115 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:37.884385 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:37.886615 3194 tablet.cc:1722] T 3fe2b75a3e59437c885abc18614f1da2 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:37.886590 3226 tablet.cc:1722] T 3fe2b75a3e59437c885abc18614f1da2 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:37.889715 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:37.892771 1169 tablet_bootstrap.cc:492] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
W20250905 19:52:37.894619 984 consensus_peers.cc:489] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:37.899744 1169 ts_tablet_manager.cc:1397] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.055s user 0.003s sys 0.006s
I20250905 19:52:37.901286 1169 raft_consensus.cc:357] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.902272 1169 raft_consensus.cc:383] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.902395 1169 raft_consensus.cc:738] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.902675 1169 consensus_queue.cc:260] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.903231 1169 ts_tablet_manager.cc:1428] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:37.903539 1169 tablet_bootstrap.cc:492] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.905309 1169 tablet_bootstrap.cc:654] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.907418 1170 tablet_bootstrap.cc:492] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:37.907595 1170 ts_tablet_manager.cc:1397] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.003s
I20250905 19:52:37.908296 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:37.908499 1170 raft_consensus.cc:357] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.908692 1170 raft_consensus.cc:383] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.908804 1170 raft_consensus.cc:738] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.916226 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:37.916941 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:37.918097 1849 raft_consensus.cc:491] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.918233 1849 raft_consensus.cc:513] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.918642 2496 raft_consensus.cc:491] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.918805 1849 leader_election.cc:290] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.918771 2496 raft_consensus.cc:513] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.919283 2496 leader_election.cc:290] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.920035 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:37.920053 711 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: fd1280fefc3e43db9928567a734644f1
I20250905 19:52:37.920212 906 raft_consensus.cc:2466] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
W20250905 19:52:37.920267 712 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: fd1280fefc3e43db9928567a734644f1
I20250905 19:52:37.920346 712 leader_election.cc:304] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:37.920624 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:37.920713 986 leader_election.cc:304] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:37.920967 2496 raft_consensus.cc:2802] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:37.921032 2496 raft_consensus.cc:491] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:37.921093 2496 raft_consensus.cc:3058] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.921130 1849 raft_consensus.cc:2747] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.921972 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:37.923283 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:37.923856 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:37.924364 984 leader_election.cc:343] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.924400 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:37.924383 2496 raft_consensus.cc:513] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.925486 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:37.925904 984 leader_election.cc:343] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.928665 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:37.929101 1170 consensus_queue.cc:260] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.929631 1170 ts_tablet_manager.cc:1428] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.022s user 0.012s sys 0.000s
I20250905 19:52:37.930104 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:37.930821 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:37.930999 906 raft_consensus.cc:3058] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:37.931142 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:37.931327 1522 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:37.933686 906 raft_consensus.cc:2466] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:37.933720 1517 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet fbba1ad28a614b06afe865c478901a32 consensus state has no leader
I20250905 19:52:37.933897 1517 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 58) took 9642 ms. Trace:
I20250905 19:52:37.934298 986 leader_election.cc:304] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.933996 1517 rpcz_store.cc:276] 0905 19:52:28.291126 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.032713 (+7741587us) service_pool.cc:225] Handling call
0905 19:52:36.106056 (+ 73343us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:36.106294 (+ 238us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:36.106519 (+ 225us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:36.106613 (+ 94us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:36.821688 (+715075us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:37.933865 (+1112177us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.210090 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.210232 (+ 142us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.210240 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:36.210242 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:36.210368 (+ 126us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.210383 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.210388 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:36.210469 (+ 81us) tablet.cc:699] Row locks acquired
0905 19:52:36.210471 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.210508 (+ 37us) write_op.cc:270] Start()
0905 19:52:36.210548 (+ 40us) write_op.cc:276] Timestamp: P: 1757101956210492 usec, L: 0
0905 19:52:36.210551 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.309944 (+ 99393us) spinlock_profiling.cc:243] Waited 99.1 ms on lock 0x616000052cb8. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fabe30e0675 00007fabe31aa33b 00007fabe3185358 00007fabe3184c05 00007fabe452d2d5 00007fabe452bdf9 00007fabe4534220 00007fabe453402d 0000000000a774dc 00007fabd773314f
0905 19:52:36.310175 (+ 231us) log.cc:844] Serialized 405 byte log entry
0905 19:52:36.316903 (+ 6728us) spinlock_profiling.cc:243] Waited 6.11 ms on lock 0x616000052cb8. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe30d04cf 00007fabe30cf97e 00007fabe3183ed5 00007fabe3184d39 00007fabe452d2d5 00007fabe452bdf9 00007fabe4534220 00007fabe453402d 0000000000a774dc
0905 19:52:36.820411 (+503508us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.821374 (+ 963us) write_op.cc:301] APPLY: starting
0905 19:52:36.821384 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.821411 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.821413 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.821444 (+ 31us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.821466 (+ 22us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.821472 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:36.821521 (+ 49us) log.cc:844] Serialized 23 byte log entry
0905 19:52:36.821559 (+ 38us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.821570 (+ 11us) write_op.cc:454] Released schema lock
0905 19:52:36.821574 (+ 4us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":933,"apply.run_cpu_time_us":258,"apply.run_wall_time_us":258,"num_ops":1,"prepare.queue_time_us":62,"prepare.run_cpu_time_us":2284,"prepare.run_wall_time_us":106865,"replication_time_us":609843,"spinlock_wait_cycles":241967616,"thread_start_us":1636,"threads_started":1}]]}
I20250905 19:52:37.934602 3151 raft_consensus.cc:2802] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:37.934710 3151 raft_consensus.cc:695] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
W20250905 19:52:37.934803 1517 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:44759 (request call id 74) took 1259 ms (client timeout 1000 ms). Trace:
W20250905 19:52:37.934903 1517 rpcz_store.cc:269] 0905 19:52:36.675261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:37.934738 (+1259477us) service_pool.cc:212] Skipping call since client already timed out
0905 19:52:37.934775 (+ 37us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:52:37.934948 3151 consensus_queue.cc:237] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:37.935118 1517 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:56757 (request call id 71) took 1168 ms (client timeout 1000 ms). Trace:
W20250905 19:52:37.935215 1517 rpcz_store.cc:269] 0905 19:52:36.766415 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:37.935065 (+1168650us) service_pool.cc:212] Skipping call since client already timed out
0905 19:52:37.935095 (+ 30us) inbound_call.cc:173] Queueing failure response
Metrics: {}
I20250905 19:52:37.938280 2496 leader_election.cc:290] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:37.936355 1170 tablet_bootstrap.cc:492] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:37.939502 2838 raft_consensus.cc:491] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:37.939651 2838 raft_consensus.cc:513] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.940266 2838 leader_election.cc:290] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:37.940716 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "414f30a772524c33a74864ff741c865f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:37.941144 848 leader_election.cc:343] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.941547 1170 tablet_bootstrap.cc:654] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.944468 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "414f30a772524c33a74864ff741c865f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:37.944970 849 leader_election.cc:343] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:37.945077 849 leader_election.cc:304] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:37.945617 2413 raft_consensus.cc:2747] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:37.948732 1171 tablet_bootstrap.cc:492] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:37.948884 1171 ts_tablet_manager.cc:1397] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.099s user 0.002s sys 0.002s
I20250905 19:52:37.949613 1171 raft_consensus.cc:357] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.949847 1171 raft_consensus.cc:383] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.949921 1171 raft_consensus.cc:738] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.952152 1171 consensus_queue.cc:260] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.952781 1171 ts_tablet_manager.cc:1428] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:37.953126 1171 tablet_bootstrap.cc:492] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:37.958786 2652 ts_tablet_manager.cc:1918] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:37.959875 2652 log.cc:1199] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6221d7d058944bdb9aa730791cf2417b
I20250905 19:52:37.960386 2652 ts_tablet_manager.cc:1939] T 6221d7d058944bdb9aa730791cf2417b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.960798 1169 tablet_bootstrap.cc:492] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:37.960932 1169 ts_tablet_manager.cc:1397] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.057s user 0.008s sys 0.000s
I20250905 19:52:37.962267 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 123) took 3755 ms. Trace:
I20250905 19:52:37.962385 1169 raft_consensus.cc:357] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.974301 1169 raft_consensus.cc:383] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:37.974398 1169 raft_consensus.cc:738] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:37.974263 2652 rpcz_store.cc:276] 0905 19:52:34.207142 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.207733 (+ 591us) service_pool.cc:225] Handling call
0905 19:52:37.879577 (+3671844us) ts_tablet_manager.cc:1145] Deleting tablet 6221d7d058944bdb9aa730791cf2417b
0905 19:52:37.881092 (+ 1515us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.895480 (+ 14388us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.938871 (+ 43391us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.962248 (+ 23377us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3670741}
I20250905 19:52:37.976056 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 120) took 3782 ms. Trace:
I20250905 19:52:37.976225 2652 rpcz_store.cc:276] 0905 19:52:34.206674 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.208400 (+ 1726us) service_pool.cc:225] Handling call
0905 19:52:37.975929 (+3767529us) ts_tablet_manager.cc:1145] Deleting tablet a8b6823580cb4040b49ed0a49b748965
0905 19:52:37.975940 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.976038 (+ 98us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3766858}
I20250905 19:52:37.976538 2496 consensus_queue.cc:1035] T e5fc2c2ba49444af939f5a975165362b P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:37.976699 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 122) took 3769 ms. Trace:
I20250905 19:52:37.976779 2652 rpcz_store.cc:276] 0905 19:52:34.207023 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.208436 (+ 1413us) service_pool.cc:225] Handling call
0905 19:52:37.976616 (+3768180us) ts_tablet_manager.cc:1145] Deleting tablet 06850faa25de4b3d83f6bf98ccd370f0
0905 19:52:37.976625 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.976683 (+ 58us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3766136}
I20250905 19:52:37.977010 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 130) took 3766 ms. Trace:
I20250905 19:52:37.977063 2652 rpcz_store.cc:276] 0905 19:52:34.210435 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.210514 (+ 79us) service_pool.cc:225] Handling call
0905 19:52:37.976948 (+3766434us) ts_tablet_manager.cc:1145] Deleting tablet 44769f1170354870ae2d34fdb67ffbae
0905 19:52:37.976955 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.976998 (+ 43us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3766322}
I20250905 19:52:37.977435 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 131) took 3766 ms. Trace:
I20250905 19:52:37.977563 2652 rpcz_store.cc:276] 0905 19:52:34.211122 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.212907 (+ 1785us) service_pool.cc:225] Handling call
0905 19:52:37.977359 (+3764452us) ts_tablet_manager.cc:1145] Deleting tablet c46ea498d6a14d1ebdffea415cfc7114
0905 19:52:37.977367 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.977422 (+ 55us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3764270}
I20250905 19:52:37.977873 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 132) took 3766 ms. Trace:
I20250905 19:52:37.977988 2652 rpcz_store.cc:276] 0905 19:52:34.211270 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.213121 (+ 1851us) service_pool.cc:225] Handling call
0905 19:52:37.977809 (+3764688us) ts_tablet_manager.cc:1145] Deleting tablet 8503f6918ffb44efa36214633c93e31d
0905 19:52:37.977816 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.977861 (+ 45us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3764579}
I20250905 19:52:37.978294 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 127) took 3768 ms. Trace:
I20250905 19:52:37.978399 2652 rpcz_store.cc:276] 0905 19:52:34.210105 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.214049 (+ 3944us) service_pool.cc:225] Handling call
0905 19:52:37.978230 (+3764181us) ts_tablet_manager.cc:1145] Deleting tablet 091ef981d6b54756830927ea111e78aa
0905 19:52:37.978238 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.978282 (+ 44us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3762582}
I20250905 19:52:37.978727 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 129) took 3768 ms. Trace:
I20250905 19:52:37.978873 2652 rpcz_store.cc:276] 0905 19:52:34.210340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.214077 (+ 3737us) service_pool.cc:225] Handling call
0905 19:52:37.978660 (+3764583us) ts_tablet_manager.cc:1145] Deleting tablet ab4f69c45295422f85b310aa55220e9b
0905 19:52:37.978668 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.978714 (+ 46us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3760776}
I20250905 19:52:37.980226 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6221d7d058944bdb9aa730791cf2417b (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]) successfully deleted
W20250905 19:52:37.980559 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a8b6823580cb4040b49ed0a49b748965 because the tablet was not found. No further retry: Not found: Tablet not found: a8b6823580cb4040b49ed0a49b748965
I20250905 19:52:37.980597 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 128) took 3770 ms. Trace:
W20250905 19:52:37.980788 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 06850faa25de4b3d83f6bf98ccd370f0 because the tablet was not found. No further retry: Not found: Tablet not found: 06850faa25de4b3d83f6bf98ccd370f0
I20250905 19:52:37.980765 2652 rpcz_store.cc:276] 0905 19:52:34.210230 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:34.214018 (+ 3788us) service_pool.cc:225] Handling call
0905 19:52:37.980518 (+3766500us) ts_tablet_manager.cc:1145] Deleting tablet ba161282480e4736889559a2e37b195f
0905 19:52:37.980528 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.980582 (+ 54us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":3760074}
W20250905 19:52:37.981047 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 44769f1170354870ae2d34fdb67ffbae because the tablet was not found. No further retry: Not found: Tablet not found: 44769f1170354870ae2d34fdb67ffbae
I20250905 19:52:37.981205 2652 tablet_replica.cc:331] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
W20250905 19:52:37.981259 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet c46ea498d6a14d1ebdffea415cfc7114 because the tablet was not found. No further retry: Not found: Tablet not found: c46ea498d6a14d1ebdffea415cfc7114
W20250905 19:52:37.981460 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 8503f6918ffb44efa36214633c93e31d because the tablet was not found. No further retry: Not found: Tablet not found: 8503f6918ffb44efa36214633c93e31d
I20250905 19:52:37.981516 2652 raft_consensus.cc:2241] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:37.981775 2652 raft_consensus.cc:2270] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:37.983238 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ab4f69c45295422f85b310aa55220e9b because the tablet was not found. No further retry: Not found: Tablet not found: ab4f69c45295422f85b310aa55220e9b
W20250905 19:52:37.983484 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ba161282480e4736889559a2e37b195f because the tablet was not found. No further retry: Not found: Tablet not found: ba161282480e4736889559a2e37b195f
I20250905 19:52:37.983899 2652 ts_tablet_manager.cc:1905] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:37.974647 1169 consensus_queue.cc:260] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:37.990257 1169 ts_tablet_manager.cc:1428] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.029s user 0.002s sys 0.000s
I20250905 19:52:37.990562 1169 tablet_bootstrap.cc:492] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:37.992260 1169 tablet_bootstrap.cc:654] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:37.996304 2652 ts_tablet_manager.cc:1918] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:37.996490 2652 log.cc:1199] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/dc95a83ca872495894ef439cdda4e364
I20250905 19:52:37.996860 2652 ts_tablet_manager.cc:1939] T dc95a83ca872495894ef439cdda4e364 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:37.997676 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 134) took 2047 ms. Trace:
I20250905 19:52:37.997828 2652 rpcz_store.cc:276] 0905 19:52:35.950678 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.950872 (+ 194us) service_pool.cc:225] Handling call
0905 19:52:37.981100 (+2030228us) ts_tablet_manager.cc:1145] Deleting tablet dc95a83ca872495894ef439cdda4e364
0905 19:52:37.981109 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.993318 (+ 12209us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.996209 (+ 2891us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:37.997654 (+ 1445us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1866128}
I20250905 19:52:37.998315 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 138) took 2037 ms. Trace:
I20250905 19:52:37.998454 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet dc95a83ca872495894ef439cdda4e364 (table default.Table_90ee22819c074028aaf36f24d7de572d [id=8b3dde81ffd64ad8ac94ab82fe25b8ef]) successfully deleted
I20250905 19:52:37.998463 2652 rpcz_store.cc:276] 0905 19:52:35.960628 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.960934 (+ 306us) service_pool.cc:225] Handling call
0905 19:52:37.998214 (+2037280us) ts_tablet_manager.cc:1145] Deleting tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff
0905 19:52:37.998226 (+ 12us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.998299 (+ 73us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1883170}
I20250905 19:52:37.999027 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 139) took 2038 ms. Trace:
W20250905 19:52:37.999153 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet c0ba1d4a7cd444c688d5a9bc8fd1a0ff because the tablet was not found. No further retry: Not found: Tablet not found: c0ba1d4a7cd444c688d5a9bc8fd1a0ff
I20250905 19:52:37.999159 2652 rpcz_store.cc:276] 0905 19:52:35.960766 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.960856 (+ 90us) service_pool.cc:225] Handling call
0905 19:52:37.998945 (+2038089us) ts_tablet_manager.cc:1145] Deleting tablet 5bd9380e8ffc432da4a684934c2df68f
0905 19:52:37.998954 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.999013 (+ 59us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1883857}
I20250905 19:52:37.999583 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 136) took 2039 ms. Trace:
W20250905 19:52:37.999671 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 5bd9380e8ffc432da4a684934c2df68f because the tablet was not found. No further retry: Not found: Tablet not found: 5bd9380e8ffc432da4a684934c2df68f
I20250905 19:52:37.999711 2652 rpcz_store.cc:276] 0905 19:52:35.960372 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.961195 (+ 823us) service_pool.cc:225] Handling call
0905 19:52:37.999508 (+2038313us) ts_tablet_manager.cc:1145] Deleting tablet 1f1a8e7059b348cba34069ef49bf1aad
0905 19:52:37.999516 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:37.999568 (+ 52us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884374}
I20250905 19:52:38.000241 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 137) took 2039 ms. Trace:
W20250905 19:52:38.000295 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 1f1a8e7059b348cba34069ef49bf1aad because the tablet was not found. No further retry: Not found: Tablet not found: 1f1a8e7059b348cba34069ef49bf1aad
I20250905 19:52:38.000401 2652 rpcz_store.cc:276] 0905 19:52:35.960506 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.960847 (+ 341us) service_pool.cc:225] Handling call
0905 19:52:38.000095 (+2039248us) ts_tablet_manager.cc:1145] Deleting tablet a5af8232d31240ff855b79dcea29396b
0905 19:52:38.000103 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:38.000226 (+ 123us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884928}
I20250905 19:52:38.000792 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 142) took 2036 ms. Trace:
W20250905 19:52:38.000896 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet a5af8232d31240ff855b79dcea29396b because the tablet was not found. No further retry: Not found: Tablet not found: a5af8232d31240ff855b79dcea29396b
I20250905 19:52:38.000917 2652 rpcz_store.cc:276] 0905 19:52:35.964671 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.964968 (+ 297us) service_pool.cc:225] Handling call
0905 19:52:38.000720 (+2035752us) ts_tablet_manager.cc:1145] Deleting tablet ac5c88cf8d15433a940e937f0a225582
0905 19:52:38.000728 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:38.000778 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1885517}
I20250905 19:52:38.001377 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 144) took 2021 ms. Trace:
W20250905 19:52:38.001415 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet ac5c88cf8d15433a940e937f0a225582 because the tablet was not found. No further retry: Not found: Tablet not found: ac5c88cf8d15433a940e937f0a225582
I20250905 19:52:38.001528 2652 rpcz_store.cc:276] 0905 19:52:35.979459 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:35.979526 (+ 67us) service_pool.cc:225] Handling call
0905 19:52:38.001302 (+2021776us) ts_tablet_manager.cc:1145] Deleting tablet 8d38452d3a0641ffa4af5d935f8ad9f0
0905 19:52:38.001312 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:38.001362 (+ 50us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1886058}
I20250905 19:52:38.001930 2652 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:39172 (request call id 147) took 1999 ms. Trace:
W20250905 19:52:38.002023 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 8d38452d3a0641ffa4af5d935f8ad9f0 because the tablet was not found. No further retry: Not found: Tablet not found: 8d38452d3a0641ffa4af5d935f8ad9f0
I20250905 19:52:38.002050 2652 rpcz_store.cc:276] 0905 19:52:36.002010 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.002065 (+ 55us) service_pool.cc:225] Handling call
0905 19:52:38.001860 (+1999795us) ts_tablet_manager.cc:1145] Deleting tablet 8edf4b54974c427a855eee5792d75292
0905 19:52:38.001868 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:38.001916 (+ 48us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1884518}
I20250905 19:52:38.000659 1171 tablet_bootstrap.cc:654] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:38.002570 1479 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 8edf4b54974c427a855eee5792d75292 because the tablet was not found. No further retry: Not found: Tablet not found: 8edf4b54974c427a855eee5792d75292
I20250905 19:52:38.016911 1170 tablet_bootstrap.cc:492] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.017079 1170 ts_tablet_manager.cc:1397] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.081s user 0.001s sys 0.016s
I20250905 19:52:38.017805 1170 raft_consensus.cc:357] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.017988 1170 raft_consensus.cc:383] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.018040 1170 raft_consensus.cc:738] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.018275 1170 consensus_queue.cc:260] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.018822 1170 ts_tablet_manager.cc:1428] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:38.019106 1170 tablet_bootstrap.cc:492] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.022435 1170 tablet_bootstrap.cc:654] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.039057 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:38.045795 1170 tablet_bootstrap.cc:492] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.045959 1170 ts_tablet_manager.cc:1397] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.027s user 0.002s sys 0.006s
I20250905 19:52:38.046634 1170 raft_consensus.cc:357] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.046809 1170 raft_consensus.cc:383] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.046873 1170 raft_consensus.cc:738] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.047071 1170 consensus_queue.cc:260] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.047508 1170 ts_tablet_manager.cc:1428] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:38.047797 1170 tablet_bootstrap.cc:492] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.049338 1170 tablet_bootstrap.cc:654] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.060856 1169 tablet_bootstrap.cc:492] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.061017 1169 ts_tablet_manager.cc:1397] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.071s user 0.007s sys 0.004s
I20250905 19:52:38.061673 1169 raft_consensus.cc:357] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.061843 1169 raft_consensus.cc:383] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.061892 1169 raft_consensus.cc:738] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.062100 1169 consensus_queue.cc:260] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.062915 1169 ts_tablet_manager.cc:1428] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:38.063263 1169 tablet_bootstrap.cc:492] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.063373 1517 catalog_manager.cc:5582] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.065124 1169 tablet_bootstrap.cc:654] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.065431 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:38.068056 1517 catalog_manager.cc:5582] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.074527 2838 raft_consensus.cc:491] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.074688 2838 raft_consensus.cc:513] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.075264 2838 leader_election.cc:290] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.076215 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.076864 848 leader_election.cc:343] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.077672 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.078344 849 leader_election.cc:343] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.078452 849 leader_election.cc:304] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.078804 2789 consensus_queue.cc:1035] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.079404 2789 raft_consensus.cc:2747] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.086881 2413 consensus_queue.cc:1035] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.089396 1517 catalog_manager.cc:5582] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.095026 1169 tablet_bootstrap.cc:492] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.095250 1169 ts_tablet_manager.cc:1397] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.000s
I20250905 19:52:38.095957 1169 raft_consensus.cc:357] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.096239 1169 raft_consensus.cc:383] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.096247 1170 tablet_bootstrap.cc:492] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.096354 1169 raft_consensus.cc:738] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.096400 1170 ts_tablet_manager.cc:1397] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.049s user 0.003s sys 0.003s
I20250905 19:52:38.096685 1169 consensus_queue.cc:260] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.097285 1169 ts_tablet_manager.cc:1428] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:38.097623 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:38.098109 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:38.098062 1170 raft_consensus.cc:357] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.098340 1170 raft_consensus.cc:383] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.098428 1170 raft_consensus.cc:738] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.098937 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]). Expected version 2 got 0
I20250905 19:52:38.099682 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:38.100097 1517 catalog_manager.cc:5582] T 2568b9f6f37b48efbe92cef4c19d5dfb P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.101006 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:38.101354 1517 catalog_manager.cc:5582] T e7086c5bfa4644d08e4f55231b85acf4 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.102313 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:38.103547 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:38.105692 1170 consensus_queue.cc:260] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.107622 1170 ts_tablet_manager.cc:1428] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:38.108109 1170 tablet_bootstrap.cc:492] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.109396 2413 raft_consensus.cc:491] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.109112 1517 catalog_manager.cc:5582] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.109537 2413 raft_consensus.cc:513] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.109812 1517 catalog_manager.cc:5582] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:38.110159 2413 leader_election.cc:290] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.110237 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:38.110512 1517 catalog_manager.cc:5582] T 95b32fff7d4745c79185a5d894ef3342 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.111037 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bd731a303a545f492b19f774520582c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.111061 1517 catalog_manager.cc:5582] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.111354 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:38.111568 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
W20250905 19:52:38.111652 848 leader_election.cc:343] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.111790 1517 catalog_manager.cc:5582] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.112104 1169 tablet_bootstrap.cc:492] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.112202 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:38.112282 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bd731a303a545f492b19f774520582c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.113095 849 leader_election.cc:343] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.112999 1517 catalog_manager.cc:5582] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.113215 849 leader_election.cc:304] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.113746 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:38.113901 1169 tablet_bootstrap.cc:654] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.113880 2789 raft_consensus.cc:2747] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.113958 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]). Expected version 1 got 0
I20250905 19:52:38.114421 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:38.115139 1517 catalog_manager.cc:5582] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.115459 1517 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:38.121992 1171 tablet_bootstrap.cc:492] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.122141 1171 ts_tablet_manager.cc:1397] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.169s user 0.022s sys 0.026s
I20250905 19:52:38.122807 1171 raft_consensus.cc:357] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.122951 1171 raft_consensus.cc:383] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.122990 1171 raft_consensus.cc:738] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.123184 1171 consensus_queue.cc:260] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.123633 1171 ts_tablet_manager.cc:1428] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
W20250905 19:52:38.124629 1542 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 55cecf59d3f24db88bb10b8e12c12f82 consensus state has no leader
I20250905 19:52:38.124821 1542 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 55) took 9843 ms. Trace:
I20250905 19:52:38.124931 1542 rpcz_store.cc:276] 0905 19:52:28.281420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.025472 (+7744052us) service_pool.cc:225] Handling call
0905 19:52:36.025924 (+ 452us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:36.026140 (+ 216us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:36.055847 (+ 29707us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:36.055959 (+ 112us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:36.817254 (+761295us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:38.124789 (+1307535us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.057358 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.057458 (+ 100us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.057463 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:36.057465 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:36.057683 (+ 218us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.057695 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.057699 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:36.057773 (+ 74us) tablet.cc:699] Row locks acquired
0905 19:52:36.057777 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:36.057818 (+ 41us) write_op.cc:270] Start()
0905 19:52:36.057855 (+ 37us) write_op.cc:276] Timestamp: P: 1757101956057801 usec, L: 0
0905 19:52:36.057859 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.058026 (+ 167us) log.cc:844] Serialized 405 byte log entry
0905 19:52:36.815657 (+757631us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.816511 (+ 854us) write_op.cc:301] APPLY: starting
0905 19:52:36.816524 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.816562 (+ 38us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.816590 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.816637 (+ 47us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.816669 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.816678 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:36.816742 (+ 64us) log.cc:844] Serialized 23 byte log entry
0905 19:52:36.816787 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.816801 (+ 14us) write_op.cc:454] Released schema lock
0905 19:52:36.816807 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":816,"apply.run_cpu_time_us":354,"apply.run_wall_time_us":354,"num_ops":1,"prepare.queue_time_us":53,"prepare.run_cpu_time_us":681,"prepare.run_wall_time_us":1985,"replication_time_us":757781,"thread_start_us":4471,"threads_started":1}]]}
I20250905 19:52:38.125937 1542 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_3e897c08965d4265b2a5937b15f5c715"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:38.126593 1542 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3e897c08965d4265b2a5937b15f5c715 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.132925 1170 tablet_bootstrap.cc:654] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.134068 3151 raft_consensus.cc:491] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.134204 3151 raft_consensus.cc:513] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.134794 3151 leader_election.cc:290] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.135386 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a860ab3f5ce54bd2b75bb71730a3c6e6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.135567 906 raft_consensus.cc:2466] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.135951 986 leader_election.cc:304] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.136219 3151 raft_consensus.cc:2802] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.136282 3151 raft_consensus.cc:491] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.136349 3151 raft_consensus.cc:3058] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.136409 1169 tablet_bootstrap.cc:492] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.136509 1169 ts_tablet_manager.cc:1397] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.024s user 0.003s sys 0.000s
I20250905 19:52:38.137193 1169 raft_consensus.cc:357] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.137343 1169 raft_consensus.cc:383] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.137388 1169 raft_consensus.cc:738] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.137566 1169 consensus_queue.cc:260] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.138726 821 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:38.138882 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:38.138937 3151 raft_consensus.cc:513] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.139314 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a860ab3f5ce54bd2b75bb71730a3c6e6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.139600 1098 heartbeater.cc:344] Connected to a master server at 127.0.104.60:34395
I20250905 19:52:38.139741 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
W20250905 19:52:38.139814 984 leader_election.cc:343] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.142380 2496 raft_consensus.cc:491] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.142508 2496 raft_consensus.cc:513] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.143805 769 raft_consensus.cc:3058] T 160a7814abf9423dbe965c252c20fa7f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.147047 1536 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84] alter complete (version 1)
W20250905 19:52:38.147168 1546 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 414f30a772524c33a74864ff741c865f consensus state has no leader
I20250905 19:52:38.147385 1546 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 57) took 9859 ms. Trace:
I20250905 19:52:38.148006 1543 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 56e7ab709fe04479a7b1f17dba2fd603
I20250905 19:52:38.147493 1546 rpcz_store.cc:276] 0905 19:52:28.288234 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.029494 (+7741260us) service_pool.cc:225] Handling call
0905 19:52:36.057718 (+ 28224us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:36.057920 (+ 202us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:36.058061 (+ 141us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:36.058131 (+ 70us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:36.821305 (+763174us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:38.147318 (+1326013us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.069709 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.069896 (+ 187us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.069907 (+ 11us) write_op.cc:435] Acquired schema lock
0905 19:52:36.069912 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:36.070046 (+ 134us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.070119 (+ 73us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.070126 (+ 7us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:36.070193 (+ 67us) tablet.cc:699] Row locks acquired
0905 19:52:36.070195 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.070235 (+ 40us) write_op.cc:270] Start()
0905 19:52:36.070269 (+ 34us) write_op.cc:276] Timestamp: P: 1757101956070218 usec, L: 0
0905 19:52:36.070272 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.070470 (+ 198us) log.cc:844] Serialized 405 byte log entry
0905 19:52:36.820355 (+749885us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.821004 (+ 649us) write_op.cc:301] APPLY: starting
0905 19:52:36.821020 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.821047 (+ 27us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.821065 (+ 18us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.821104 (+ 39us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.821129 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.821135 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:36.821185 (+ 50us) log.cc:844] Serialized 23 byte log entry
0905 19:52:36.821224 (+ 39us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.821236 (+ 12us) write_op.cc:454] Released schema lock
0905 19:52:36.821240 (+ 4us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:36.821312 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.821317 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":615,"apply.run_cpu_time_us":302,"apply.run_wall_time_us":301,"num_ops":1,"prepare.queue_time_us":11149,"prepare.run_cpu_time_us":766,"prepare.run_wall_time_us":1186,"replication_time_us":750061}]]}
I20250905 19:52:38.148447 1546 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_c6be55785e1244cb85764306f76c04a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:38.148612 1543 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 60) took 8781 ms. Trace:
W20250905 19:52:38.149055 1546 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c6be55785e1244cb85764306f76c04a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.148692 1543 rpcz_store.cc:276] 0905 19:52:29.367133 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.464827 (+7097694us) service_pool.cc:225] Handling call
0905 19:52:36.465315 (+ 488us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:36.465738 (+ 423us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:36.465819 (+ 81us) catalog_manager.cc:2799] Locking tablets
0905 19:52:36.819222 (+353403us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:36.935683 (+116461us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:36.935779 (+ 96us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:38.147944 (+1212165us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:38.148545 (+ 601us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.820783 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.820957 (+ 174us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.820965 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:36.820969 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:36.821151 (+ 182us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.821166 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.821170 (+ 4us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:36.821297 (+ 127us) tablet.cc:699] Row locks acquired
0905 19:52:36.821301 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:36.821350 (+ 49us) write_op.cc:270] Start()
0905 19:52:36.821388 (+ 38us) write_op.cc:276] Timestamp: P: 1757101956821331 usec, L: 0
0905 19:52:36.821393 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.877881 (+ 56488us) log.cc:844] Serialized 725 byte log entry
0905 19:52:36.879425 (+ 1544us) spinlock_profiling.cc:243] Waited 56.2 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fabe3184d62 00007fabe452d2d5 00007fabe452bdf9 00007fabe4534220 00007fabe453402d 0000000000a774dc 00007fabd773314f 00007fabd773641b 00007fabd7736298 0000000000a774dc
0905 19:52:36.934876 (+ 55451us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.935029 (+ 153us) write_op.cc:301] APPLY: starting
0905 19:52:36.935059 (+ 30us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.935138 (+ 79us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.935142 (+ 4us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.935240 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.935292 (+ 52us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:36.935304 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:36.935422 (+ 118us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.935546 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.935574 (+ 28us) write_op.cc:454] Released schema lock
0905 19:52:36.935585 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:36.935691 (+ 106us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.935698 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":47,"apply.run_cpu_time_us":661,"apply.run_wall_time_us":662,"num_ops":2,"prepare.queue_time_us":549,"prepare.run_cpu_time_us":1357,"prepare.run_wall_time_us":58692,"replication_time_us":113415,"spinlock_wait_cycles":129295232,"thread_start_us":3935,"threads_started":1}]]}
I20250905 19:52:38.149530 2422 consensus_queue.cc:1035] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.149526 1543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_6e40598f9dbc40ac9860eb5b78130a93"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:38.150049 1543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6e40598f9dbc40ac9860eb5b78130a93 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.150542 1540 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:56416 (request call id 56) took 9865 ms. Trace:
I20250905 19:52:38.150974 3063 raft_consensus.cc:491] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.150661 1540 rpcz_store.cc:276] 0905 19:52:28.284974 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.028079 (+7743105us) service_pool.cc:225] Handling call
0905 19:52:36.055936 (+ 27857us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:36.056220 (+ 284us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:36.057059 (+ 839us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:36.057143 (+ 84us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:36.821141 (+763998us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:38.150496 (+1329355us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.059363 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.059443 (+ 80us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.059448 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:36.059449 (+ 1us) tablet.cc:660] Decoding operations
0905 19:52:36.059538 (+ 89us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.059547 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.059552 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:36.059597 (+ 45us) tablet.cc:699] Row locks acquired
0905 19:52:36.059599 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:36.059627 (+ 28us) write_op.cc:270] Start()
0905 19:52:36.059652 (+ 25us) write_op.cc:276] Timestamp: P: 1757101956059614 usec, L: 0
0905 19:52:36.059656 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.059789 (+ 133us) log.cc:844] Serialized 405 byte log entry
0905 19:52:36.820261 (+760472us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.820572 (+ 311us) write_op.cc:301] APPLY: starting
0905 19:52:36.820586 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.820623 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.820656 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.820707 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.820739 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:36.820745 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:36.820807 (+ 62us) log.cc:844] Serialized 23 byte log entry
0905 19:52:36.820864 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.820878 (+ 14us) write_op.cc:454] Released schema lock
0905 19:52:36.820883 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:36.820944 (+ 61us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.820948 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":249,"apply.run_cpu_time_us":366,"apply.run_wall_time_us":365,"num_ops":1,"prepare.queue_time_us":1702,"prepare.run_cpu_time_us":535,"prepare.run_wall_time_us":10277,"replication_time_us":760577}]]}
I20250905 19:52:38.151101 3063 raft_consensus.cc:513] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.151408 1540 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_d090d4e81b7c45e49c707734cfeae659"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:38.151669 3063 leader_election.cc:290] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.151811 1171 tablet_bootstrap.cc:492] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:38.151947 1540 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d090d4e81b7c45e49c707734cfeae659 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.152272 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.152940 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 56e7ab709fe04479a7b1f17dba2fd603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:36 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:38.153162 2376 tablet_replica.cc:331] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:38.153352 2376 raft_consensus.cc:2241] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:38.153470 1539 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 8feaf4ec6b0e4ce9909dd72258912935
I20250905 19:52:38.153602 2376 pending_rounds.cc:70] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Trying to abort 1 pending ops.
I20250905 19:52:38.153645 2376 pending_rounds.cc:77] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089616100978688 op_type: NO_OP noop_request { }
I20250905 19:52:38.153725 2376 raft_consensus.cc:2887] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:38.153805 2376 raft_consensus.cc:2270] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.153960 1539 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:56416 (request call id 59) took 8813 ms. Trace:
I20250905 19:52:38.154071 1171 tablet_bootstrap.cc:654] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.154353 2376 ts_tablet_manager.cc:1905] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.154062 1539 rpcz_store.cc:276] 0905 19:52:29.340898 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:36.036965 (+6696067us) service_pool.cc:225] Handling call
0905 19:52:36.058821 (+ 21856us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:36.058960 (+ 139us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:36.059035 (+ 75us) catalog_manager.cc:2799] Locking tablets
0905 19:52:36.820480 (+761445us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:36.939177 (+118697us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:36.939277 (+ 100us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:38.153404 (+1214127us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:38.153926 (+ 522us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:36.879539 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:36.879768 (+ 229us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:36.879778 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:36.879782 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:36.880096 (+ 314us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:36.880111 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:36.880120 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:36.880266 (+ 146us) tablet.cc:699] Row locks acquired
0905 19:52:36.880270 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:36.880317 (+ 47us) write_op.cc:270] Start()
0905 19:52:36.880363 (+ 46us) write_op.cc:276] Timestamp: P: 1757101956880297 usec, L: 0
0905 19:52:36.880369 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:36.880588 (+ 219us) log.cc:844] Serialized 759 byte log entry
0905 19:52:36.934971 (+ 54383us) op_driver.cc:464] REPLICATION: finished
0905 19:52:36.935792 (+ 821us) write_op.cc:301] APPLY: starting
0905 19:52:36.935806 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:36.935852 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:36.935888 (+ 36us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:36.935954 (+ 66us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:36.936009 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:36.936021 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:36.936119 (+ 98us) log.cc:844] Serialized 29 byte log entry
0905 19:52:36.936169 (+ 50us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.936823 (+ 654us) write_op.cc:454] Released schema lock
0905 19:52:36.936833 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:36.936921 (+ 88us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:36.936928 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":780,"apply.run_cpu_time_us":491,"apply.run_wall_time_us":1115,"num_ops":2,"prepare.queue_time_us":58419,"prepare.run_cpu_time_us":956,"prepare.run_wall_time_us":1501,"replication_time_us":54581}]]}
I20250905 19:52:38.154803 1539 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_00d9a0127df4444eb081316b26c6ec87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:38.155274 1539 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_00d9a0127df4444eb081316b26c6ec87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.158619 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 8feaf4ec6b0e4ce9909dd72258912935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:36 UTC) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:38.160264 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 56e7ab709fe04479a7b1f17dba2fd603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:36 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:38.161227 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 8feaf4ec6b0e4ce9909dd72258912935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:36 UTC) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:38.164791 1542 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 61) took 8585 ms. Trace:
I20250905 19:52:38.165290 1542 rpcz_store.cc:276] 0905 19:52:29.579420 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.125709 (+8546289us) service_pool.cc:225] Handling call
0905 19:52:38.126696 (+ 987us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.127192 (+ 496us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.164549 (+ 37357us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.164655 (+ 106us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.164750 (+ 95us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.142694 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.142830 (+ 136us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.142838 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:38.142840 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.142953 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.142967 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.142972 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.143072 (+ 100us) tablet.cc:699] Row locks acquired
0905 19:52:38.143075 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:38.143113 (+ 38us) write_op.cc:270] Start()
0905 19:52:38.143153 (+ 40us) write_op.cc:276] Timestamp: P: 1757101958143097 usec, L: 0
0905 19:52:38.143158 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.143388 (+ 230us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.153278 (+ 9890us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.158585 (+ 5307us) write_op.cc:301] APPLY: starting
0905 19:52:38.158604 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.158654 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.158676 (+ 22us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.158796 (+ 120us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.158831 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.158838 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:38.158930 (+ 92us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.159003 (+ 73us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.159027 (+ 24us) write_op.cc:454] Released schema lock
0905 19:52:38.159034 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:38.159106 (+ 72us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.159110 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":5236,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":518,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":820,"prepare.run_wall_time_us":820,"raft.queue_time_us":1230,"raft.run_cpu_time_us":546,"raft.run_wall_time_us":545,"replication_time_us":10094}]]}
I20250905 19:52:38.166756 1542 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_ec95e016850b4e4eb1105ac761a31b74"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:38.167438 1542 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ec95e016850b4e4eb1105ac761a31b74 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.170645 1543 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 63) took 8561 ms. Trace:
I20250905 19:52:38.170782 1543 rpcz_store.cc:276] 0905 19:52:29.609327 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.149303 (+8539976us) service_pool.cc:225] Handling call
0905 19:52:38.150139 (+ 836us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.150613 (+ 474us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.170319 (+ 19706us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.170466 (+ 147us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.170572 (+ 106us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.159935 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.160060 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.160065 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:38.160067 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.160140 (+ 73us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.160150 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.160157 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.160222 (+ 65us) tablet.cc:699] Row locks acquired
0905 19:52:38.160224 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:38.160254 (+ 30us) write_op.cc:270] Start()
0905 19:52:38.160282 (+ 28us) write_op.cc:276] Timestamp: P: 1757101958160241 usec, L: 0
0905 19:52:38.160287 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.160437 (+ 150us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.167804 (+ 7367us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.168598 (+ 794us) write_op.cc:301] APPLY: starting
0905 19:52:38.168611 (+ 13us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.168647 (+ 36us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.168679 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.168765 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.168790 (+ 25us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.168796 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:38.168847 (+ 51us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.168896 (+ 49us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.168911 (+ 15us) write_op.cc:454] Released schema lock
0905 19:52:38.168916 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":760,"apply.run_cpu_time_us":370,"apply.run_wall_time_us":370,"num_ops":2,"prepare.queue_time_us":8878,"prepare.run_cpu_time_us":553,"prepare.run_wall_time_us":552,"replication_time_us":7501,"threads_started":1}]]}
I20250905 19:52:38.177276 1546 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 62) took 8575 ms. Trace:
I20250905 19:52:38.177312 2376 ts_tablet_manager.cc:1918] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:38.177675 2376 log.cc:1199] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/56e7ab709fe04479a7b1f17dba2fd603
I20250905 19:52:38.177404 1546 rpcz_store.cc:276] 0905 19:52:29.601533 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.148202 (+8546669us) service_pool.cc:225] Handling call
0905 19:52:38.149143 (+ 941us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.149656 (+ 513us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.176957 (+ 27301us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.177076 (+ 119us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.177208 (+ 132us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.159247 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.159358 (+ 111us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.159366 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:38.159370 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:38.159485 (+ 115us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.159499 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.159506 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.159583 (+ 77us) tablet.cc:699] Row locks acquired
0905 19:52:38.159585 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:38.159621 (+ 36us) write_op.cc:270] Start()
0905 19:52:38.159651 (+ 30us) write_op.cc:276] Timestamp: P: 1757101958159607 usec, L: 0
0905 19:52:38.159654 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.159829 (+ 175us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.167742 (+ 7913us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.168026 (+ 284us) write_op.cc:301] APPLY: starting
0905 19:52:38.168042 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.168102 (+ 60us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.168144 (+ 42us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.168246 (+ 102us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.168278 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.168285 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:38.168372 (+ 87us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.168439 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.168454 (+ 15us) write_op.cc:454] Released schema lock
0905 19:52:38.168460 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:38.168530 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.168533 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":188,"apply.run_cpu_time_us":495,"apply.run_wall_time_us":494,"num_ops":2,"prepare.queue_time_us":9103,"prepare.run_cpu_time_us":650,"prepare.run_wall_time_us":649,"replication_time_us":8062}]]}
I20250905 19:52:38.178028 1546 master_service.cc:948] ReplaceTablet: received request to replace tablet edb9379335ca4eb5a32a6cbe85601a1f from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:38.178434 2413 consensus_queue.cc:1035] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.179199 2376 ts_tablet_manager.cc:1939] T 56e7ab709fe04479a7b1f17dba2fd603 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:38.180119 1547 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7] alter complete (version 1)
I20250905 19:52:38.184265 2422 raft_consensus.cc:491] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.184417 2422 raft_consensus.cc:513] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.185024 2422 leader_election.cc:290] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.187259 1849 consensus_queue.cc:1035] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.188239 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edb9379335ca4eb5a32a6cbe85601a1f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.188514 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 56e7ab709fe04479a7b1f17dba2fd603 (table default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]) successfully deleted
I20250905 19:52:38.188877 2376 tablet_replica.cc:331] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:38.188933 848 leader_election.cc:343] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.189195 2376 raft_consensus.cc:2241] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:38.189543 2376 raft_consensus.cc:2270] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.190121 2376 ts_tablet_manager.cc:1905] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:38.197690 1472 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 414f30a772524c33a74864ff741c865f consensus state has no leader
I20250905 19:52:38.201021 1171 tablet_bootstrap.cc:492] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.201179 1171 ts_tablet_manager.cc:1397] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.049s user 0.006s sys 0.014s
I20250905 19:52:38.201920 1171 raft_consensus.cc:357] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.202139 1171 raft_consensus.cc:383] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.202200 1171 raft_consensus.cc:738] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.202574 1171 consensus_queue.cc:260] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.203063 1171 ts_tablet_manager.cc:1428] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:38.204002 1171 tablet_bootstrap.cc:492] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.205560 1171 tablet_bootstrap.cc:654] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.209677 1170 tablet_bootstrap.cc:492] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.209947 1170 ts_tablet_manager.cc:1397] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.102s user 0.003s sys 0.000s
I20250905 19:52:38.212884 1170 raft_consensus.cc:357] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.213184 1170 raft_consensus.cc:383] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.213294 1170 raft_consensus.cc:738] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.213553 1171 tablet_bootstrap.cc:492] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.213696 1171 ts_tablet_manager.cc:1397] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20250905 19:52:38.213582 1170 consensus_queue.cc:260] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.214665 1171 raft_consensus.cc:357] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.214891 1171 raft_consensus.cc:383] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.214958 1171 raft_consensus.cc:738] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.215204 1171 consensus_queue.cc:260] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.215844 1171 ts_tablet_manager.cc:1428] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:38.216197 1171 tablet_bootstrap.cc:492] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.217769 1170 ts_tablet_manager.cc:1428] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:38.218413 1171 tablet_bootstrap.cc:654] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.218753 1170 tablet_bootstrap.cc:492] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.219462 2376 ts_tablet_manager.cc:1918] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:38.220140 3063 consensus_queue.cc:1035] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.221194 1170 tablet_bootstrap.cc:654] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.152614 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "56e7ab709fe04479a7b1f17dba2fd603" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.223452 3151 leader_election.cc:290] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.223618 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 56e7ab709fe04479a7b1f17dba2fd603 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:36 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:38.224242 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 8feaf4ec6b0e4ce9909dd72258912935 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:36 UTC) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:38.223630 1042 raft_consensus.cc:2391] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:38.225207 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edb9379335ca4eb5a32a6cbe85601a1f" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.225769 2496 leader_election.cc:290] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.228286 2376 log.cc:1199] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/8feaf4ec6b0e4ce9909dd72258912935
I20250905 19:52:38.228415 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.228679 2376 ts_tablet_manager.cc:1939] T 8feaf4ec6b0e4ce9909dd72258912935 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:38.227418 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.228876 907 raft_consensus.cc:2466] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.229348 986 leader_election.cc:304] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.229899 3151 raft_consensus.cc:2802] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.229997 3151 raft_consensus.cc:491] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.230077 3151 raft_consensus.cc:3058] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.230962 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 8feaf4ec6b0e4ce9909dd72258912935 (table default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]) successfully deleted
I20250905 19:52:38.233350 3151 raft_consensus.cc:513] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.233781 1171 tablet_bootstrap.cc:492] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.233902 1171 ts_tablet_manager.cc:1397] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20250905 19:52:38.234369 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a860ab3f5ce54bd2b75bb71730a3c6e6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:38.234640 1171 raft_consensus.cc:357] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.234797 1171 raft_consensus.cc:383] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.234844 1171 raft_consensus.cc:738] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.235087 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.235054 1171 consensus_queue.cc:260] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.235237 907 raft_consensus.cc:3058] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.235599 1171 ts_tablet_manager.cc:1428] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:38.235829 1171 tablet_bootstrap.cc:492] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.237466 1171 tablet_bootstrap.cc:654] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.237872 907 raft_consensus.cc:2466] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.238420 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a860ab3f5ce54bd2b75bb71730a3c6e6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.238615 906 raft_consensus.cc:3058] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.241961 906 raft_consensus.cc:2466] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.242729 986 leader_election.cc:304] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.243116 986 leader_election.cc:304] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.243403 2496 raft_consensus.cc:2802] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.243535 2496 raft_consensus.cc:695] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.243566 3253 tablet_replica.cc:331] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:38.243755 3253 raft_consensus.cc:2241] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.243840 2496 consensus_queue.cc:237] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.243884 3253 raft_consensus.cc:2270] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.244992 2931 raft_consensus.cc:2802] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.245113 2931 raft_consensus.cc:695] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.245330 2931 consensus_queue.cc:237] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.245723 1171 tablet_bootstrap.cc:492] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.245838 1171 ts_tablet_manager.cc:1397] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.002s
I20250905 19:52:38.246577 1171 raft_consensus.cc:357] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.246729 1171 raft_consensus.cc:383] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.246783 1171 raft_consensus.cc:738] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.246977 1171 consensus_queue.cc:260] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:38.247349 849 leader_election.cc:343] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.247445 849 leader_election.cc:304] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.247450 1171 ts_tablet_manager.cc:1428] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:38.247745 1171 tablet_bootstrap.cc:492] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.247958 3155 consensus_queue.cc:1035] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.248102 2422 raft_consensus.cc:2747] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.248327 2998 raft_consensus.cc:491] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.248437 2998 raft_consensus.cc:513] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.249006 2998 leader_election.cc:290] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.249447 1171 tablet_bootstrap.cc:654] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:38.249882 984 leader_election.cc:343] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.250198 984 leader_election.cc:343] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.251065 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180437d1534946619384307ddcb7c937" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.251515 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:38.252983 711 leader_election.cc:304] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:38.253228 711 raft_consensus.cc:2684] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:38.253918 3063 raft_consensus.cc:491] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.254029 3063 raft_consensus.cc:513] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.254530 3063 leader_election.cc:290] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.258085 1169 ts_tablet_manager.cc:1428] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.121s user 0.001s sys 0.001s
I20250905 19:52:38.258255 2652 tablet_replica.cc:331] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:38.258471 2652 raft_consensus.cc:2241] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.258636 2652 raft_consensus.cc:2270] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.259230 2652 ts_tablet_manager.cc:1905] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.260072 1169 tablet_bootstrap.cc:492] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.261163 3151 leader_election.cc:290] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.261615 1171 tablet_bootstrap.cc:492] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.261754 1171 ts_tablet_manager.cc:1397] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.000s
I20250905 19:52:38.262790 1171 raft_consensus.cc:357] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.262972 1171 raft_consensus.cc:383] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.263024 1171 raft_consensus.cc:738] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:38.263309 984 leader_election.cc:343] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.263491 984 leader_election.cc:343] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.263245 1171 consensus_queue.cc:260] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.263957 1171 ts_tablet_manager.cc:1428] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:38.264247 1171 tablet_bootstrap.cc:492] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.266150 1171 tablet_bootstrap.cc:654] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:38.271059 984 consensus_peers.cc:489] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:38.276139 1171 tablet_bootstrap.cc:492] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.276265 1171 ts_tablet_manager.cc:1397] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250905 19:52:38.277099 1171 raft_consensus.cc:357] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.277362 1171 raft_consensus.cc:383] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.277420 1171 raft_consensus.cc:738] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.277711 1171 consensus_queue.cc:260] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.278450 1171 ts_tablet_manager.cc:1428] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:38.278700 1171 tablet_bootstrap.cc:492] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.279469 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180437d1534946619384307ddcb7c937" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.280424 907 raft_consensus.cc:2466] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.280800 1171 tablet_bootstrap.cc:654] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.280802 1522 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73] alter complete (version 1)
I20250905 19:52:38.281098 986 leader_election.cc:304] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.281605 3175 raft_consensus.cc:2802] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.281685 3175 raft_consensus.cc:491] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.282395 1540 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 64) took 8653 ms. Trace:
I20250905 19:52:38.282547 1540 rpcz_store.cc:276] 0905 19:52:29.629119 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.151209 (+8522090us) service_pool.cc:225] Handling call
0905 19:52:38.152054 (+ 845us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.152488 (+ 434us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.282079 (+129591us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.282190 (+ 111us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.282340 (+ 150us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.161038 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.161119 (+ 81us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.161125 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:38.161127 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.161194 (+ 67us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.161201 (+ 7us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.161207 (+ 6us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.161269 (+ 62us) tablet.cc:699] Row locks acquired
0905 19:52:38.161272 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:38.161298 (+ 26us) write_op.cc:270] Start()
0905 19:52:38.161320 (+ 22us) write_op.cc:276] Timestamp: P: 1757101958161284 usec, L: 0
0905 19:52:38.161323 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.161465 (+ 142us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.246160 (+ 84695us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.280658 (+ 34498us) write_op.cc:301] APPLY: starting
0905 19:52:38.280669 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.280700 (+ 31us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.280729 (+ 29us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.280869 (+ 140us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.280903 (+ 34us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.280911 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:38.280976 (+ 65us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.281042 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.281057 (+ 15us) write_op.cc:454] Released schema lock
0905 19:52:38.281065 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":34429,"apply.run_cpu_time_us":467,"apply.run_wall_time_us":467,"num_ops":2,"prepare.queue_time_us":8174,"prepare.run_cpu_time_us":470,"prepare.run_wall_time_us":469,"replication_time_us":84798}]]}
I20250905 19:52:38.285547 3268 raft_consensus.cc:491] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.285655 3268 raft_consensus.cc:513] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.286172 3268 leader_election.cc:290] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.286587 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.287109 848 leader_election.cc:343] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.289666 848 consensus_peers.cc:489] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:38.292091 1171 tablet_bootstrap.cc:492] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.292246 1171 ts_tablet_manager.cc:1397] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20250905 19:52:38.291947 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.293236 1171 raft_consensus.cc:357] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.293419 1171 raft_consensus.cc:383] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.293480 1171 raft_consensus.cc:738] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.280143 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3d3a7b03d5e34f8dbcd5b6b994c9b419" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.294232 711 leader_election.cc:304] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:38.294765 3175 raft_consensus.cc:3058] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.294996 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.295873 849 leader_election.cc:343] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.296048 849 leader_election.cc:304] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.296062 1171 consensus_queue.cc:260] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.296378 2738 raft_consensus.cc:2747] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.296644 3268 raft_consensus.cc:491] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.296770 3268 raft_consensus.cc:513] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.297343 3268 leader_election.cc:290] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.297534 3063 raft_consensus.cc:2747] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.297694 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d1d24868204ce09a7f5e40357fec18" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.298017 1543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_b24d9f16949145cda6b4f82d78993fa0"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:38.298339 848 leader_election.cc:343] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.296651 1171 ts_tablet_manager.cc:1428] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20250905 19:52:38.298606 1543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b24d9f16949145cda6b4f82d78993fa0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.298767 1171 tablet_bootstrap.cc:492] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.299052 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d1d24868204ce09a7f5e40357fec18" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.299351 3175 raft_consensus.cc:513] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:38.299594 849 leader_election.cc:343] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.299691 849 leader_election.cc:304] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.300038 3268 raft_consensus.cc:2747] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.300458 1171 tablet_bootstrap.cc:654] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.300787 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180437d1534946619384307ddcb7c937" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:38.302896 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "180437d1534946619384307ddcb7c937" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.303074 906 raft_consensus.cc:3058] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.305603 906 raft_consensus.cc:2466] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.306221 986 leader_election.cc:304] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.314147 2413 raft_consensus.cc:491] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.314311 2413 raft_consensus.cc:513] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.314931 2413 leader_election.cc:290] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.315704 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.316248 848 leader_election.cc:343] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.317629 1171 tablet_bootstrap.cc:492] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.317793 1171 ts_tablet_manager.cc:1397] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.004s
I20250905 19:52:38.318692 1171 raft_consensus.cc:357] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.318858 1171 raft_consensus.cc:383] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.318907 1171 raft_consensus.cc:738] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.319088 1171 consensus_queue.cc:260] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.319578 1171 ts_tablet_manager.cc:1428] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:38.320218 1171 tablet_bootstrap.cc:492] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:38.321795 984 leader_election.cc:343] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.323205 1171 tablet_bootstrap.cc:654] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.323274 3151 raft_consensus.cc:2802] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.324362 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.324968 849 leader_election.cc:343] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.325110 849 leader_election.cc:304] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.325474 3268 raft_consensus.cc:2747] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.326336 3175 leader_election.cc:290] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.326599 3268 raft_consensus.cc:491] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.326709 3268 raft_consensus.cc:513] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.327306 3268 leader_election.cc:290] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.327615 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a5d38136c034b8497045176f749a98d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.328197 848 leader_election.cc:343] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.327255 1169 tablet_bootstrap.cc:654] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.328135 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a5d38136c034b8497045176f749a98d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.328708 849 leader_election.cc:343] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.328814 849 leader_election.cc:304] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.329219 3268 raft_consensus.cc:2747] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.330868 1169 tablet_bootstrap.cc:492] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.331072 1169 ts_tablet_manager.cc:1397] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.071s user 0.004s sys 0.000s
I20250905 19:52:38.331912 1169 raft_consensus.cc:357] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.332145 1169 raft_consensus.cc:383] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.332204 3151 raft_consensus.cc:695] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.332579 3151 consensus_queue.cc:237] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.335187 2652 ts_tablet_manager.cc:1918] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:38.335348 2652 log.cc:1199] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/56e7ab709fe04479a7b1f17dba2fd603
I20250905 19:52:38.335738 2652 ts_tablet_manager.cc:1939] T 56e7ab709fe04479a7b1f17dba2fd603 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:38.336437 2652 tablet_replica.cc:331] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:38.336555 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 56e7ab709fe04479a7b1f17dba2fd603 (table default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]) successfully deleted
I20250905 19:52:38.336632 2652 raft_consensus.cc:2241] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.336776 2652 raft_consensus.cc:2270] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.344940 3274 raft_consensus.cc:491] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.345106 3274 raft_consensus.cc:513] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.345729 3274 leader_election.cc:290] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.346587 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.346711 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.346860 907 raft_consensus.cc:2466] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.347090 2652 ts_tablet_manager.cc:1905] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.347196 986 leader_election.cc:304] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.347488 3274 raft_consensus.cc:2802] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.347581 3274 raft_consensus.cc:491] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.347640 3274 raft_consensus.cc:3058] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.350257 3274 raft_consensus.cc:513] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.350764 3274 leader_election.cc:290] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:38.347469 984 leader_election.cc:343] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.351539 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:38.352536 984 leader_election.cc:343] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.353432 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.353848 906 raft_consensus.cc:3058] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.356036 3063 raft_consensus.cc:491] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.356266 3063 raft_consensus.cc:513] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.356952 3063 leader_election.cc:290] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.357803 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.358305 907 raft_consensus.cc:2391] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:38.360576 1171 tablet_bootstrap.cc:492] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.360747 1171 ts_tablet_manager.cc:1397] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.041s user 0.004s sys 0.001s
I20250905 19:52:38.361713 1171 raft_consensus.cc:357] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.361914 1171 raft_consensus.cc:383] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.361986 1171 raft_consensus.cc:738] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.362210 1171 consensus_queue.cc:260] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.362862 1171 ts_tablet_manager.cc:1428] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:38.363200 1171 tablet_bootstrap.cc:492] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.371088 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "91396422d0bb468395db04aa80262882" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.371412 3253 ts_tablet_manager.cc:1905] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.371767 711 leader_election.cc:304] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:38.372557 3063 raft_consensus.cc:3058] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.373847 1547 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0] alter complete (version 1)
I20250905 19:52:38.375761 3063 raft_consensus.cc:2747] T 91396422d0bb468395db04aa80262882 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.376904 3274 raft_consensus.cc:491] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.377118 3274 raft_consensus.cc:513] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.377889 3274 leader_election.cc:290] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.378582 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.379084 984 leader_election.cc:343] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.381769 1522 catalog_manager.cc:6127] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e] alter complete (version 1)
I20250905 19:52:38.389403 1171 tablet_bootstrap.cc:654] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.391467 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.391700 907 raft_consensus.cc:2466] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.392340 986 leader_election.cc:304] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.393038 3274 raft_consensus.cc:2802] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.393118 3274 raft_consensus.cc:491] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.393175 3274 raft_consensus.cc:3058] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.396044 3274 raft_consensus.cc:513] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.397209 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:38.397800 984 leader_election.cc:343] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.400745 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.400931 907 raft_consensus.cc:3058] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.402300 3274 leader_election.cc:290] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.404009 907 raft_consensus.cc:2466] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.404286 3274 raft_consensus.cc:491] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.404520 3274 raft_consensus.cc:513] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.404731 986 leader_election.cc:304] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.405517 3151 raft_consensus.cc:2802] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.405529 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c77dd068ea5c495ca8aa0e7a6823d792" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.405750 3151 raft_consensus.cc:695] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.414978 1543 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 67) took 8739 ms. Trace:
I20250905 19:52:38.415279 1170 tablet_bootstrap.cc:492] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.415470 1170 ts_tablet_manager.cc:1397] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.197s user 0.036s sys 0.040s
I20250905 19:52:38.415113 1543 rpcz_store.cc:276] 0905 19:52:29.675315 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.172062 (+8496747us) service_pool.cc:225] Handling call
0905 19:52:38.297935 (+125873us) spinlock_profiling.cc:243] Waited 126 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaec6eb3 00007fabe1cfe7a6 00007fabe1cfe4d5 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:38.298742 (+ 807us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.299262 (+ 520us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.385682 (+ 86420us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.414796 (+ 29114us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.414911 (+ 115us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.303254 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.303413 (+ 159us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.303420 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:38.303424 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:38.303551 (+ 127us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.303566 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.303574 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.303665 (+ 91us) tablet.cc:699] Row locks acquired
0905 19:52:38.303667 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:38.303700 (+ 33us) write_op.cc:270] Start()
0905 19:52:38.303739 (+ 39us) write_op.cc:276] Timestamp: P: 1757101958303686 usec, L: 0
0905 19:52:38.303746 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.303981 (+ 235us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.384845 (+ 80864us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.385024 (+ 179us) write_op.cc:301] APPLY: starting
0905 19:52:38.385036 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.385070 (+ 34us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.385304 (+ 234us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.385423 (+ 119us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.385458 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.385467 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:38.385533 (+ 66us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.385585 (+ 52us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.385603 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:38.385608 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:38.387585 (+ 1977us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.387591 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":288732928,"child_traces":[["op",{"apply.queue_time_us":121,"apply.run_cpu_time_us":457,"apply.run_wall_time_us":774,"num_ops":2,"prepare.queue_time_us":3457,"prepare.run_cpu_time_us":818,"prepare.run_wall_time_us":818,"replication_time_us":80984}]]}
I20250905 19:52:38.415987 1543 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56416:
name: "default.Table_6a2d2733af51406295e2d14f82f6cb1e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:38.416498 1170 raft_consensus.cc:357] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:38.416656 1543 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6a2d2733af51406295e2d14f82f6cb1e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:38.416715 1170 raft_consensus.cc:383] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.416778 1170 raft_consensus.cc:738] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.417001 1170 consensus_queue.cc:260] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.417927 1170 ts_tablet_manager.cc:1428] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250905 19:52:38.418298 1170 tablet_bootstrap.cc:492] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.418629 1542 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 66) took 8753 ms. Trace:
I20250905 19:52:38.418738 1542 rpcz_store.cc:276] 0905 19:52:29.665631 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.166530 (+8500899us) service_pool.cc:225] Handling call
0905 19:52:38.167529 (+ 999us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.168023 (+ 494us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.371052 (+203029us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.418499 (+ 47447us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.418595 (+ 96us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.294577 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.294775 (+ 198us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.294784 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:38.294789 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:38.294938 (+ 149us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.294955 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.294963 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.295080 (+ 117us) tablet.cc:699] Row locks acquired
0905 19:52:38.295084 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:38.295133 (+ 49us) write_op.cc:270] Start()
0905 19:52:38.295197 (+ 64us) write_op.cc:276] Timestamp: P: 1757101958295113 usec, L: 0
0905 19:52:38.295204 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.295442 (+ 238us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.369513 (+ 74071us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.370531 (+ 1018us) write_op.cc:301] APPLY: starting
0905 19:52:38.370545 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.370592 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.370635 (+ 43us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.370758 (+ 123us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.370797 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.370808 (+ 11us) write_op.cc:312] APPLY: finished
0905 19:52:38.370867 (+ 59us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.370915 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.370932 (+ 17us) write_op.cc:454] Released schema lock
0905 19:52:38.370939 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":988,"apply.run_cpu_time_us":461,"apply.run_wall_time_us":462,"num_ops":2,"prepare.queue_time_us":116942,"prepare.run_cpu_time_us":944,"prepare.run_wall_time_us":943,"replication_time_us":74294}]]}
I20250905 19:52:38.419317 1542 master_service.cc:948] ReplaceTablet: received request to replace tablet 9477afb5b47c46d4b24128a6c4bdabc6 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:38.420960 1539 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 65) took 8783 ms. Trace:
I20250905 19:52:38.421069 1539 rpcz_store.cc:276] 0905 19:52:29.637663 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.154616 (+8516953us) service_pool.cc:225] Handling call
0905 19:52:38.155357 (+ 741us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.155766 (+ 409us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.370446 (+214680us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.420821 (+ 50375us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.420920 (+ 99us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.169574 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.169686 (+ 112us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.169690 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:38.169692 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.169821 (+ 129us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.169835 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.169842 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.169928 (+ 86us) tablet.cc:699] Row locks acquired
0905 19:52:38.169933 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:38.169971 (+ 38us) write_op.cc:270] Start()
0905 19:52:38.170006 (+ 35us) write_op.cc:276] Timestamp: P: 1757101958169954 usec, L: 0
0905 19:52:38.170012 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.294217 (+124205us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.294440 (+ 223us) spinlock_profiling.cc:243] Waited 124 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fabe3184d62 00007fabe452d2d5 00007fabe452bdf9 00007fabe4534220 00007fabe453402d 0000000000a774dc 00007fabd773314f 00007fabd773641b 00007fabd7736298 0000000000a774dc
0905 19:52:38.369442 (+ 75002us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.369937 (+ 495us) write_op.cc:301] APPLY: starting
0905 19:52:38.369953 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.370004 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.370009 (+ 5us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.370138 (+ 129us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.370174 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.370180 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:38.370270 (+ 90us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.370339 (+ 69us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.370359 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:38.370365 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:38.370460 (+ 95us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.370464 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":421,"apply.run_cpu_time_us":518,"apply.run_wall_time_us":519,"num_ops":2,"prepare.queue_time_us":3960,"prepare.run_cpu_time_us":2549,"prepare.run_wall_time_us":124907,"replication_time_us":199386,"spinlock_wait_cycles":285170432}]]}
I20250905 19:52:38.422075 1539 master_service.cc:948] ReplaceTablet: received request to replace tablet 42f3c1efb94e446fb0128b6b412821e2 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:38.422470 1539 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 71) took 8472 ms. Trace:
I20250905 19:52:38.422554 1539 rpcz_store.cc:276] 0905 19:52:29.949523 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.422004 (+8472481us) service_pool.cc:225] Handling call
0905 19:52:38.422416 (+ 412us) inbound_call.cc:173] Queueing success response
Metrics: {}
I20250905 19:52:38.424700 1539 master_service.cc:948] ReplaceTablet: received request to replace tablet e6ac13b19fb6427a9ef35a5160033a54 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:38.426869 1171 tablet_bootstrap.cc:492] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.426995 1171 ts_tablet_manager.cc:1397] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.064s user 0.003s sys 0.007s
I20250905 19:52:38.427731 1171 raft_consensus.cc:357] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.427894 1171 raft_consensus.cc:383] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.427943 1171 raft_consensus.cc:738] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.433444 1170 tablet_bootstrap.cc:654] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.434737 959 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:38.432931 1171 consensus_queue.cc:260] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.441404 1171 ts_tablet_manager.cc:1428] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250905 19:52:38.441700 1171 tablet_bootstrap.cc:492] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.443295 1171 tablet_bootstrap.cc:654] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.456673 1546 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:38.457299 1546 catalog_manager.cc:6652] ReplaceTablet: tablet edb9379335ca4eb5a32a6cbe85601a1f deleted and replaced by tablet fe00969eb9654976a0efeac2fa7c91d3
I20250905 19:52:38.457444 1546 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 68) took 8749 ms. Trace:
I20250905 19:52:38.457561 1546 rpcz_store.cc:276] 0905 19:52:29.708130 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.177938 (+8469808us) service_pool.cc:225] Handling call
0905 19:52:38.310175 (+132237us) spinlock_profiling.cc:243] Waited 132 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 00007fabe64f314d 00007fabe64f3243 00007fabe31b2941 00007fabead3da7d 00007fabeaed3bfb 00007fabe1d0b0e7 00007fabe1d0ae15 00007fabde4d60be 00007fabde4d4f9e 00007fabde4db1d5 00007fabde4dbaff
0905 19:52:38.457401 (+147226us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.405641 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.405814 (+ 173us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.405821 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:38.405823 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.405966 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.405977 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.405982 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.406066 (+ 84us) tablet.cc:699] Row locks acquired
0905 19:52:38.406071 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:38.406111 (+ 40us) write_op.cc:270] Start()
0905 19:52:38.406150 (+ 39us) write_op.cc:276] Timestamp: P: 1757101958406094 usec, L: 0
0905 19:52:38.406153 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.409800 (+ 3647us) log.cc:844] Serialized 622 byte log entry
0905 19:52:38.453032 (+ 43232us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.455936 (+ 2904us) write_op.cc:301] APPLY: starting
0905 19:52:38.455947 (+ 11us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.456014 (+ 67us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.456052 (+ 38us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.456160 (+ 108us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.456224 (+ 64us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:38.456235 (+ 11us) write_op.cc:312] APPLY: finished
0905 19:52:38.456303 (+ 68us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.456360 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.456378 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:38.456386 (+ 8us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"spinlock_wait_cycles":302972288,"child_traces":[["op",{"apply.queue_time_us":2873,"apply.run_cpu_time_us":520,"apply.run_wall_time_us":520,"num_ops":2,"prepare.queue_time_us":14071,"prepare.run_cpu_time_us":885,"prepare.run_wall_time_us":4432,"replication_time_us":46864,"threads_started":1}]]}
I20250905 19:52:38.461040 882 tablet_service.cc:1515] Processing DeleteTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f with delete_type TABLET_DATA_DELETED (replaced by tablet fe00969eb9654976a0efeac2fa7c91d3) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:38.461265 2376 tablet_replica.cc:331] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:38.461458 2376 raft_consensus.cc:2241] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.461594 2376 raft_consensus.cc:2270] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.462077 2376 ts_tablet_manager.cc:1905] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.465879 751 tablet_service.cc:1515] Processing DeleteTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f with delete_type TABLET_DATA_DELETED (replaced by tablet fe00969eb9654976a0efeac2fa7c91d3) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:38.406213 984 leader_election.cc:343] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.470628 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 64430.9us
I20250905 19:52:38.470693 2376 ts_tablet_manager.cc:1918] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:38.470798 2376 log.cc:1199] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:38.475478 2496 raft_consensus.cc:491] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.475653 2496 raft_consensus.cc:513] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.480275 2998 raft_consensus.cc:491] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.480424 2376 ts_tablet_manager.cc:1939] T edb9379335ca4eb5a32a6cbe85601a1f P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:38.480402 2998 raft_consensus.cc:513] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.480959 2998 leader_election.cc:290] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.481019 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet edb9379335ca4eb5a32a6cbe85601a1f (table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]) successfully deleted
I20250905 19:52:38.481513 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "923ad8e40a07409e8f6a0152a0b2ef75" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.481987 984 leader_election.cc:343] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.482631 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "923ad8e40a07409e8f6a0152a0b2ef75" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.482797 909 raft_consensus.cc:2466] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.483397 986 leader_election.cc:304] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.483665 3175 raft_consensus.cc:2802] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.483754 3175 raft_consensus.cc:491] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.483836 3175 raft_consensus.cc:3058] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.484513 3292 tablet.cc:1722] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:38.485332 3253 ts_tablet_manager.cc:1918] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:38.485445 3253 log.cc:1199] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/56e7ab709fe04479a7b1f17dba2fd603
I20250905 19:52:38.485719 3253 ts_tablet_manager.cc:1939] T 56e7ab709fe04479a7b1f17dba2fd603 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:38.486409 3175 raft_consensus.cc:513] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.486824 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 56e7ab709fe04479a7b1f17dba2fd603 (table default.Table_e065514fff894d6c837edd542a939abd [id=e34cb751407241e5b33b9700cfa7cc34]) successfully deleted
I20250905 19:52:38.486912 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c77dd068ea5c495ca8aa0e7a6823d792" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.487072 909 raft_consensus.cc:2466] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.487437 3175 leader_election.cc:290] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.487457 2413 raft_consensus.cc:491] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.487562 2413 raft_consensus.cc:513] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.487668 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "923ad8e40a07409e8f6a0152a0b2ef75" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:38.488161 984 leader_election.cc:343] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.488466 986 leader_election.cc:304] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.488631 2413 leader_election.cc:290] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.488986 3253 tablet_replica.cc:331] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:38.489109 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.489168 3253 raft_consensus.cc:2241] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.489305 3253 raft_consensus.cc:2270] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
W20250905 19:52:38.489740 848 leader_election.cc:343] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.489837 3253 ts_tablet_manager.cc:1905] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.489897 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65368306f86845008d4d486023e1fc29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.489991 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.490322 984 leader_election.cc:343] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.490438 849 leader_election.cc:343] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.490537 849 leader_election.cc:304] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.490803 2496 leader_election.cc:290] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.490929 2496 raft_consensus.cc:2802] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.490983 2496 raft_consensus.cc:491] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.491039 2496 raft_consensus.cc:3058] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.491142 2738 raft_consensus.cc:2747] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.492333 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65368306f86845008d4d486023e1fc29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.492408 3274 leader_election.cc:290] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.492488 904 raft_consensus.cc:2466] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.492867 986 leader_election.cc:304] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.493453 3274 raft_consensus.cc:2802] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.493703 3274 raft_consensus.cc:491] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.493948 3274 raft_consensus.cc:3058] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.494462 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "923ad8e40a07409e8f6a0152a0b2ef75" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.494603 909 raft_consensus.cc:3058] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.496235 1169 raft_consensus.cc:738] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.497421 909 raft_consensus.cc:2466] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.498044 986 leader_election.cc:304] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.499305 1169 consensus_queue.cc:260] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.499902 1169 ts_tablet_manager.cc:1428] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.169s user 0.015s sys 0.003s
I20250905 19:52:38.500267 1169 tablet_bootstrap.cc:492] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.500692 3175 raft_consensus.cc:2802] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.508067 1169 tablet_bootstrap.cc:654] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.509099 3291 tablet.cc:1722] T 55cecf59d3f24db88bb10b8e12c12f82 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:38.510246 3296 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:38.511199 3175 raft_consensus.cc:695] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.511575 3175 consensus_queue.cc:237] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.521066 3253 ts_tablet_manager.cc:1918] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:38.523056 3253 log.cc:1199] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/8feaf4ec6b0e4ce9909dd72258912935
I20250905 19:52:38.525246 3253 ts_tablet_manager.cc:1939] T 8feaf4ec6b0e4ce9909dd72258912935 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:38.526988 1472 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 8feaf4ec6b0e4ce9909dd72258912935 (table default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]) successfully deleted
I20250905 19:52:38.530805 1171 tablet_bootstrap.cc:492] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.530783 3274 raft_consensus.cc:513] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.530954 1171 ts_tablet_manager.cc:1397] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.089s user 0.002s sys 0.004s
W20250905 19:52:38.531551 1472 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet edb9379335ca4eb5a32a6cbe85601a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edb9379335ca4eb5a32a6cbe85601a1f already in progress: creating tablet
I20250905 19:52:38.531733 1171 raft_consensus.cc:357] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.531905 1171 raft_consensus.cc:383] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.531951 1171 raft_consensus.cc:738] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.532215 1171 consensus_queue.cc:260] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.532812 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65368306f86845008d4d486023e1fc29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.533013 909 raft_consensus.cc:3058] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.531555 2496 raft_consensus.cc:513] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.534233 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c77dd068ea5c495ca8aa0e7a6823d792" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:38.534404 904 raft_consensus.cc:3058] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.534478 2496 leader_election.cc:290] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.536221 909 raft_consensus.cc:2466] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.536870 904 raft_consensus.cc:2466] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.538918 1543 rpcz_store.cc:275] Call kudu.master.MasterService.CreateTable from 127.0.0.1:56416 (request call id 69) took 8819 ms. Trace:
I20250905 19:52:38.539037 1543 rpcz_store.cc:276] 0905 19:52:29.719352 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.415745 (+8696393us) service_pool.cc:225] Handling call
0905 19:52:38.416752 (+ 1007us) catalog_manager.cc:2076] Acquired catalog manager lock
0905 19:52:38.417453 (+ 701us) catalog_manager.cc:2122] Created new table and tablet info
0905 19:52:38.536904 (+119451us) catalog_manager.cc:2182] Wrote table and tablets to system table
0905 19:52:38.538742 (+ 1838us) catalog_manager.cc:2216] Inserted table and tablets into CatalogManager maps
0905 19:52:38.538856 (+ 114us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.423055 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.423170 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.423176 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:38.423178 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.423282 (+ 104us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.423294 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.423302 (+ 8us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.423377 (+ 75us) tablet.cc:699] Row locks acquired
0905 19:52:38.423380 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:38.423426 (+ 46us) write_op.cc:270] Start()
0905 19:52:38.423462 (+ 36us) write_op.cc:276] Timestamp: P: 1757101958423399 usec, L: 0
0905 19:52:38.423465 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.423639 (+ 174us) log.cc:844] Serialized 433 byte log entry
0905 19:52:38.535573 (+111934us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.535701 (+ 128us) write_op.cc:301] APPLY: starting
0905 19:52:38.535719 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.535769 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.535822 (+ 53us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.536032 (+ 210us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.536069 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=0
0905 19:52:38.536076 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:38.536169 (+ 93us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.536236 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.536257 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:38.536266 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:38.536346 (+ 80us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.536350 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":45,"apply.run_cpu_time_us":643,"apply.run_wall_time_us":643,"num_ops":2,"prepare.queue_time_us":5111,"prepare.run_cpu_time_us":649,"prepare.run_wall_time_us":649,"replication_time_us":112078}]]}
I20250905 19:52:38.539804 1543 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_191f4f9b06184a8a8429cce5802c520e" } modify_external_catalogs: true
I20250905 19:52:38.539986 1543 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56416:
table { table_name: "default.Table_191f4f9b06184a8a8429cce5802c520e" } modify_external_catalogs: true
I20250905 19:52:38.539937 3300 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:38.544121 986 leader_election.cc:304] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:38.552150 3175 raft_consensus.cc:2802] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.552392 3175 raft_consensus.cc:695] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.552779 3175 consensus_queue.cc:237] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.553162 3274 leader_election.cc:290] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.406173 3151 consensus_queue.cc:237] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.555241 1171 ts_tablet_manager.cc:1428] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.024s user 0.000s sys 0.002s
I20250905 19:52:38.555539 1171 tablet_bootstrap.cc:492] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.557724 1171 tablet_bootstrap.cc:654] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.558701 1542 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet 9477afb5b47c46d4b24128a6c4bdabc6
I20250905 19:52:38.559295 1542 catalog_manager.cc:6652] ReplaceTablet: tablet 9477afb5b47c46d4b24128a6c4bdabc6 deleted and replaced by tablet 9ca3fc7055c44df282285806ea151e51
I20250905 19:52:38.559453 1542 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 70) took 8701 ms. Trace:
I20250905 19:52:38.448853 2652 ts_tablet_manager.cc:1918] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:38.560343 2652 log.cc:1199] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/8feaf4ec6b0e4ce9909dd72258912935
I20250905 19:52:38.560889 2652 ts_tablet_manager.cc:1939] T 8feaf4ec6b0e4ce9909dd72258912935 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:38.564580 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 9477afb5b47c46d4b24128a6c4bdabc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ca3fc7055c44df282285806ea151e51) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:38.559531 1542 rpcz_store.cc:276] 0905 19:52:29.857783 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.419259 (+8561476us) service_pool.cc:225] Handling call
0905 19:52:38.559384 (+140125us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.423726 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.423841 (+ 115us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.423845 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:38.423848 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:38.423940 (+ 92us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.423949 (+ 9us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.423956 (+ 7us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.425042 (+ 1086us) tablet.cc:699] Row locks acquired
0905 19:52:38.425045 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:38.425079 (+ 34us) write_op.cc:270] Start()
0905 19:52:38.425157 (+ 78us) write_op.cc:276] Timestamp: P: 1757101958425064 usec, L: 0
0905 19:52:38.425160 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.425299 (+ 139us) log.cc:844] Serialized 656 byte log entry
0905 19:52:38.557412 (+132113us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.557678 (+ 266us) write_op.cc:301] APPLY: starting
0905 19:52:38.557695 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.557743 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.557792 (+ 49us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.557934 (+ 142us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.558004 (+ 70us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:38.558011 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:38.558116 (+ 105us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.558199 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.558228 (+ 29us) write_op.cc:454] Released schema lock
0905 19:52:38.558238 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:38.558330 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.558337 (+ 7us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":33,"apply.run_cpu_time_us":637,"apply.run_wall_time_us":636,"num_ops":2,"prepare.queue_time_us":3384,"prepare.run_cpu_time_us":637,"prepare.run_wall_time_us":1630,"replication_time_us":132207}]]}
I20250905 19:52:38.564939 2376 tablet_replica.cc:331] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:38.565155 2376 raft_consensus.cc:2241] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:38.565443 2376 raft_consensus.cc:2270] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.565611 1542 master_service.cc:948] ReplaceTablet: received request to replace tablet d1a11777cf2d4ea9a5cdf792f73aff76 from {username='slave'} at 127.0.0.1:56416
I20250905 19:52:38.565963 2376 ts_tablet_manager.cc:1905] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.457759 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f with delete_type TABLET_DATA_DELETED (replaced by tablet fe00969eb9654976a0efeac2fa7c91d3) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:38.560194 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 9477afb5b47c46d4b24128a6c4bdabc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ca3fc7055c44df282285806ea151e51) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:38.572392 2652 tablet_replica.cc:331] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:38.572489 2376 ts_tablet_manager.cc:1918] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:38.572615 2376 log.cc:1199] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9477afb5b47c46d4b24128a6c4bdabc6
I20250905 19:52:38.572993 2376 ts_tablet_manager.cc:1939] T 9477afb5b47c46d4b24128a6c4bdabc6 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:38.573666 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:38.573685 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 8feaf4ec6b0e4ce9909dd72258912935 (table default.Table_b96ef31f0115467fb13cceb26c34bc99 [id=d145bc9561844af6be3c700eb9d20a69]) successfully deleted
W20250905 19:52:38.574087 1479 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet edb9379335ca4eb5a32a6cbe85601a1f because tablet deleting was already in progress. No further retry: Already present: State transition of tablet edb9379335ca4eb5a32a6cbe85601a1f already in progress: creating tablet
I20250905 19:52:38.576465 906 raft_consensus.cc:2466] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:38.574918 3295 tablet.cc:1722] T 55cecf59d3f24db88bb10b8e12c12f82 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:38.581651 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 9477afb5b47c46d4b24128a6c4bdabc6 with delete_type TABLET_DATA_DELETED (replaced by tablet 9ca3fc7055c44df282285806ea151e51) from {username='slave'} at 127.0.0.1:54786
W20250905 19:52:38.576188 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 65505.8us
I20250905 19:52:38.582974 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "65368306f86845008d4d486023e1fc29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:38.584954 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c77dd068ea5c495ca8aa0e7a6823d792" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:38.586124 984 leader_election.cc:343] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.586548 984 leader_election.cc:343] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.592072 1170 tablet_bootstrap.cc:492] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.592237 1170 ts_tablet_manager.cc:1397] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.174s user 0.005s sys 0.008s
I20250905 19:52:38.593019 1170 raft_consensus.cc:357] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.593214 1170 raft_consensus.cc:383] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.593271 1170 raft_consensus.cc:738] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.593468 1170 consensus_queue.cc:260] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.593964 1170 ts_tablet_manager.cc:1428] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:38.594254 1170 tablet_bootstrap.cc:492] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.595983 1171 tablet_bootstrap.cc:492] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.596131 1171 ts_tablet_manager.cc:1397] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.041s user 0.001s sys 0.002s
I20250905 19:52:38.596835 1171 raft_consensus.cc:357] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.597014 1171 raft_consensus.cc:383] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.597066 1171 raft_consensus.cc:738] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.598376 3268 raft_consensus.cc:491] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.598491 3268 raft_consensus.cc:513] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.598968 3268 leader_election.cc:290] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.599582 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.600358 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.600661 848 leader_election.cc:343] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.600780 849 leader_election.cc:343] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.600875 849 leader_election.cc:304] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.601121 3268 raft_consensus.cc:2747] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.603044 1170 tablet_bootstrap.cc:654] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.609248 3253 tablet_replica.cc:331] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:38.609547 3253 raft_consensus.cc:2241] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.609777 3253 raft_consensus.cc:2270] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.609779 1171 consensus_queue.cc:260] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.610291 1171 ts_tablet_manager.cc:1428] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250905 19:52:38.610564 1171 tablet_bootstrap.cc:492] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.610921 3253 ts_tablet_manager.cc:1905] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.611888 1171 tablet_bootstrap.cc:654] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.620348 771 raft_consensus.cc:3058] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.576833 2652 raft_consensus.cc:2241] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.625694 2652 raft_consensus.cc:2270] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.628805 2652 ts_tablet_manager.cc:1905] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.634716 1539 catalog_manager.cc:5869] T 00000000000000000000000000000000 P e38333af29db4bf6af41ee5a25d6701e: Sending DeleteTablet for 3 replicas of tablet e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:38.635325 1539 catalog_manager.cc:6652] ReplaceTablet: tablet e6ac13b19fb6427a9ef35a5160033a54 deleted and replaced by tablet 98db7e2a9d6d4f4d984388a493295d72
I20250905 19:52:38.635471 1539 rpcz_store.cc:275] Call kudu.master.MasterService.ReplaceTablet from 127.0.0.1:56416 (request call id 72) took 8569 ms. Trace:
I20250905 19:52:38.635676 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 with delete_type TABLET_DATA_DELETED (replaced by tablet 98db7e2a9d6d4f4d984388a493295d72) from {username='slave'} at 127.0.0.1:39172
I20250905 19:52:38.635586 1539 rpcz_store.cc:276] 0905 19:52:30.066349 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:38.424627 (+8358278us) service_pool.cc:225] Handling call
0905 19:52:38.635431 (+210804us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:38.464968 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:38.465081 (+ 113us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:38.465087 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:38.465089 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:38.465207 (+ 118us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:38.465217 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:38.465222 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:38.465300 (+ 78us) tablet.cc:699] Row locks acquired
0905 19:52:38.465302 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:38.465333 (+ 31us) write_op.cc:270] Start()
0905 19:52:38.465361 (+ 28us) write_op.cc:276] Timestamp: P: 1757101958465319 usec, L: 0
0905 19:52:38.465365 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:38.536199 (+ 70834us) log.cc:844] Serialized 622 byte log entry
0905 19:52:38.536379 (+ 180us) spinlock_profiling.cc:243] Waited 70.6 ms on lock 0x616000051754. stack: 00007fabd76ecdc6 00007fabd76ecb80 00007fabd66d43ad 0000000000972d94 0000000000972d5d 000000000097272a 00007fabe3184d62 00007fabe452d2d5 00007fabe452bdf9 00007fabe4534220 00007fabe453402d 0000000000a774dc 00007fabd773314f 00007fabd773641b 00007fabd7736298 0000000000a774dc
0905 19:52:38.612228 (+ 75849us) op_driver.cc:464] REPLICATION: finished
0905 19:52:38.616316 (+ 4088us) write_op.cc:301] APPLY: starting
0905 19:52:38.616333 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:38.616376 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:38.616379 (+ 3us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:38.616486 (+ 107us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:38.616541 (+ 55us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:38.616550 (+ 9us) write_op.cc:312] APPLY: finished
0905 19:52:38.616628 (+ 78us) log.cc:844] Serialized 29 byte log entry
0905 19:52:38.616690 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:38.616711 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:38.616717 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":4016,"apply.run_cpu_time_us":477,"apply.run_wall_time_us":477,"num_ops":2,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":1921,"prepare.run_wall_time_us":71465,"raft.queue_time_us":15152,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":4,"replication_time_us":146823,"spinlock_wait_cycles":162431232}]]}
I20250905 19:52:38.638485 882 tablet_service.cc:1515] Processing DeleteTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 with delete_type TABLET_DATA_DELETED (replaced by tablet 98db7e2a9d6d4f4d984388a493295d72) from {username='slave'} at 127.0.0.1:59592
I20250905 19:52:38.638733 2376 tablet_replica.cc:331] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:38.638939 2376 raft_consensus.cc:2241] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.639076 2376 raft_consensus.cc:2270] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.639593 2376 ts_tablet_manager.cc:1905] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.642341 1540 catalog_manager.cc:5582] T 32fa16972f2a42d39b338b833c185b28 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:38.648610 2376 ts_tablet_manager.cc:1918] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:38.648327 1540 catalog_manager.cc:5582] T 5302b79208494e48a5f2778c5f52ad01 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.648717 2376 log.cc:1199] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:38.648980 2376 ts_tablet_manager.cc:1939] T e6ac13b19fb6427a9ef35a5160033a54 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:38.648939 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:38.649462 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:38.649959 1484 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e6ac13b19fb6427a9ef35a5160033a54 (table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]) successfully deleted
I20250905 19:52:38.654064 1171 tablet_bootstrap.cc:492] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.654201 1171 ts_tablet_manager.cc:1397] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.044s user 0.002s sys 0.000s
I20250905 19:52:38.655797 751 tablet_service.cc:1515] Processing DeleteTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 with delete_type TABLET_DATA_DELETED (replaced by tablet 98db7e2a9d6d4f4d984388a493295d72) from {username='slave'} at 127.0.0.1:54786
I20250905 19:52:38.656605 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:38.658371 1171 raft_consensus.cc:357] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.658550 1171 raft_consensus.cc:383] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.658614 1171 raft_consensus.cc:738] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.659574 1170 tablet_bootstrap.cc:492] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.659696 1170 ts_tablet_manager.cc:1397] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.066s user 0.003s sys 0.001s
I20250905 19:52:38.660105 3282 raft_consensus.cc:491] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.660225 3282 raft_consensus.cc:513] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.660784 3282 leader_election.cc:290] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.660895 1170 raft_consensus.cc:357] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.661386 1170 raft_consensus.cc:383] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.661453 1170 raft_consensus.cc:738] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.661901 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.662328 1170 consensus_queue.cc:260] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.662811 1170 ts_tablet_manager.cc:1428] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:38.665869 1170 tablet_bootstrap.cc:492] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.667531 1170 tablet_bootstrap.cc:654] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.669063 1171 consensus_queue.cc:260] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.669750 1171 ts_tablet_manager.cc:1428] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20250905 19:52:38.670012 1171 tablet_bootstrap.cc:492] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.676705 1171 tablet_bootstrap.cc:654] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.677767 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:38.678103 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:38.678716 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:38.679584 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:38.679925 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:38.680706 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]). Expected version 1 got 0
I20250905 19:52:38.681500 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:38.682143 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:38.683264 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:38.683543 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 55cecf59d3f24db88bb10b8e12c12f82 (table default.Table_0d6fe65405f543d7a70643f94bf22da5 [id=46715848277847218d557157e4dfc77b]). Expected version 1 got 0
I20250905 19:52:38.684239 1540 catalog_manager.cc:5582] T ae6fe4620dda4b1296416e01f8ffee29 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.684679 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:38.689486 3151 raft_consensus.cc:491] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.689628 3151 raft_consensus.cc:513] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.690201 3151 leader_election.cc:290] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.691200 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e6ac13b19fb6427a9ef35a5160033a54" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.691843 984 leader_election.cc:343] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.696462 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a59d7fb5dc54b479d965282b1c7d24b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.696985 711 leader_election.cc:304] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:38.697448 3282 raft_consensus.cc:2747] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.699326 1170 tablet_bootstrap.cc:492] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.699460 1170 ts_tablet_manager.cc:1397] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.004s
I20250905 19:52:38.699810 1169 tablet_bootstrap.cc:492] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.700057 1169 ts_tablet_manager.cc:1397] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.200s user 0.011s sys 0.002s
I20250905 19:52:38.700134 1170 raft_consensus.cc:357] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.700294 1170 raft_consensus.cc:383] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.700340 1170 raft_consensus.cc:738] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.700515 1170 consensus_queue.cc:260] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.701100 1169 raft_consensus.cc:357] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.701323 1169 raft_consensus.cc:383] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.701434 1169 raft_consensus.cc:738] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.701681 1169 consensus_queue.cc:260] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.702219 1169 ts_tablet_manager.cc:1428] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:38.702831 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:38.703342 1169 tablet_bootstrap.cc:492] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.703665 1170 ts_tablet_manager.cc:1428] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:38.703689 986 leader_election.cc:304] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.703898 1170 tablet_bootstrap.cc:492] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.704209 3151 raft_consensus.cc:2802] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.704331 3151 raft_consensus.cc:695] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.704537 3151 consensus_queue.cc:237] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.705232 1169 tablet_bootstrap.cc:654] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.705518 1170 tablet_bootstrap.cc:654] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.706772 3282 raft_consensus.cc:491] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.706933 3282 raft_consensus.cc:513] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.707592 3282 leader_election.cc:290] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.708092 770 raft_consensus.cc:3058] T e10b4e82c4bc429b9a42db39c03a5267 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.709676 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.713515 1170 tablet_bootstrap.cc:492] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.713645 1170 ts_tablet_manager.cc:1397] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.003s
I20250905 19:52:38.713876 3297 consensus_queue.cc:1035] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.714890 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e7086c5bfa4644d08e4f55231b85acf4" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.715420 711 leader_election.cc:304] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:38.716041 3282 raft_consensus.cc:2747] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.719425 1170 raft_consensus.cc:357] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.719578 1170 raft_consensus.cc:383] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.719620 1170 raft_consensus.cc:738] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.720479 1170 consensus_queue.cc:260] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.720940 1170 ts_tablet_manager.cc:1428] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20250905 19:52:38.721203 1170 tablet_bootstrap.cc:492] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.722718 1170 tablet_bootstrap.cc:654] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.723778 986 leader_election.cc:304] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.724264 3297 raft_consensus.cc:2802] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:38.724403 3297 raft_consensus.cc:695] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:38.724635 3297 consensus_queue.cc:237] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.735723 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:38.737622 1170 tablet_bootstrap.cc:492] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.737751 1170 ts_tablet_manager.cc:1397] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.001s
I20250905 19:52:38.738914 1170 raft_consensus.cc:357] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.739081 1170 raft_consensus.cc:383] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.739125 1170 raft_consensus.cc:738] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.744903 1170 consensus_queue.cc:260] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.745221 2413 raft_consensus.cc:491] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.745385 2413 raft_consensus.cc:513] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.745903 2413 leader_election.cc:290] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.746402 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9ae8fd5fdc4b539308268ba9729cd6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.746858 848 leader_election.cc:343] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.747323 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9ae8fd5fdc4b539308268ba9729cd6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.747762 1170 ts_tablet_manager.cc:1428] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
W20250905 19:52:38.747764 849 leader_election.cc:343] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.747869 849 leader_election.cc:304] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.748088 1170 tablet_bootstrap.cc:492] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.748363 2422 raft_consensus.cc:2747] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.751281 1170 tablet_bootstrap.cc:654] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.758015 3299 consensus_queue.cc:1035] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.759895 3155 raft_consensus.cc:491] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.760066 3155 raft_consensus.cc:513] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.760762 3155 leader_election.cc:290] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.761025 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.761234 984 consensus_peers.cc:489] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:38.761498 848 leader_election.cc:343] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.762141 1170 tablet_bootstrap.cc:492] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.762279 1170 ts_tablet_manager.cc:1397] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.001s
I20250905 19:52:38.762965 1170 raft_consensus.cc:357] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.763131 1170 raft_consensus.cc:383] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.763187 1170 raft_consensus.cc:738] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.761355 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.764153 849 leader_election.cc:343] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.764278 849 leader_election.cc:304] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.764672 3155 raft_consensus.cc:2747] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.765007 1170 consensus_queue.cc:260] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.765494 1170 ts_tablet_manager.cc:1428] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:38.765748 1170 tablet_bootstrap.cc:492] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.772897 1540 catalog_manager.cc:5582] T fa5d34f9c0ee4d55ade61b30b0f4c293 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:38.773536 1540 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet b2c7906a718c46d9a76587a70d96a1f9 (table default.Table_fe3dee104cc640359210483bc31532cf [id=99384aac4e624d6db630b249c3ba2543]). Expected version 3 got 2
I20250905 19:52:38.776957 3291 tablet.cc:1722] T b2c7906a718c46d9a76587a70d96a1f9 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:38.778270 1170 tablet_bootstrap.cc:654] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.778323 2998 consensus_queue.cc:1035] T d59da39e7bcf497a93fb9174328c1814 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.781594 1169 tablet_bootstrap.cc:492] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.781749 1169 ts_tablet_manager.cc:1397] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.078s user 0.003s sys 0.000s
I20250905 19:52:38.782413 1169 raft_consensus.cc:357] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.782575 1169 raft_consensus.cc:383] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.782648 1169 raft_consensus.cc:738] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.788609 1171 tablet_bootstrap.cc:492] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.788753 1171 ts_tablet_manager.cc:1397] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.119s user 0.007s sys 0.019s
I20250905 19:52:38.789467 1171 raft_consensus.cc:357] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.789633 1171 raft_consensus.cc:383] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.789682 1171 raft_consensus.cc:738] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.789883 1171 consensus_queue.cc:260] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.790261 2652 ts_tablet_manager.cc:1918] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:38.790411 2652 log.cc:1199] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9477afb5b47c46d4b24128a6c4bdabc6
I20250905 19:52:38.790432 1171 ts_tablet_manager.cc:1428] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:38.790823 2652 ts_tablet_manager.cc:1939] T 9477afb5b47c46d4b24128a6c4bdabc6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:38.791828 1171 tablet_bootstrap.cc:492] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.792126 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9477afb5b47c46d4b24128a6c4bdabc6 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]) successfully deleted
I20250905 19:52:38.793982 1171 tablet_bootstrap.cc:654] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.794068 1169 consensus_queue.cc:260] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.794643 1169 ts_tablet_manager.cc:1428] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:38.800745 771 raft_consensus.cc:3058] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.804731 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu pid 1262
I20250905 19:52:38.804936 1169 tablet_bootstrap.cc:492] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.805289 1170 tablet_bootstrap.cc:492] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.805430 1170 ts_tablet_manager.cc:1397] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.040s user 0.003s sys 0.003s
I20250905 19:52:38.806178 1170 raft_consensus.cc:357] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.806349 1170 raft_consensus.cc:383] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.806404 1170 raft_consensus.cc:738] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.806596 1170 consensus_queue.cc:260] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.807078 1170 ts_tablet_manager.cc:1428] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:38.807359 1170 tablet_bootstrap.cc:492] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.808959 1170 tablet_bootstrap.cc:654] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.810703 1171 tablet_bootstrap.cc:492] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.810863 1171 ts_tablet_manager.cc:1397] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20250905 19:52:38.811589 1171 raft_consensus.cc:357] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.811777 1171 raft_consensus.cc:383] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.811841 1171 raft_consensus.cc:738] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.812083 1171 consensus_queue.cc:260] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.814882 1171 ts_tablet_manager.cc:1428] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20250905 19:52:38.815172 1171 tablet_bootstrap.cc:492] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.816113 2998 raft_consensus.cc:491] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.816319 2998 raft_consensus.cc:513] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.817178 2998 leader_election.cc:290] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.817358 1171 tablet_bootstrap.cc:654] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.817857 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aff2acc3f8245e1bca41f971f8e74ce" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.818455 984 leader_election.cc:343] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.820652 2652 tablet_replica.cc:331] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:38.821310 2652 raft_consensus.cc:2241] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:38.821810 2652 raft_consensus.cc:2270] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:38.824020 2652 ts_tablet_manager.cc:1905] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:38.822520 1169 tablet_bootstrap.cc:654] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.851297 1171 tablet_bootstrap.cc:492] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.851437 1171 ts_tablet_manager.cc:1397] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.036s user 0.000s sys 0.010s
I20250905 19:52:38.852169 1171 raft_consensus.cc:357] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.852336 1171 raft_consensus.cc:383] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.852391 1171 raft_consensus.cc:738] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.855305 1171 consensus_queue.cc:260] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.856252 1171 ts_tablet_manager.cc:1428] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20250905 19:52:38.856535 1171 tablet_bootstrap.cc:492] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.858232 1171 tablet_bootstrap.cc:654] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.860672 1631 generic_service.cc:196] Checking for leaks (request via RPC)
I20250905 19:52:38.871336 3282 raft_consensus.cc:491] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.871524 3282 raft_consensus.cc:513] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.872285 3282 leader_election.cc:290] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.876032 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.876114 2652 ts_tablet_manager.cc:1918] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:38.876219 2652 log.cc:1199] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:38.876735 2652 ts_tablet_manager.cc:1939] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:38.877560 1479 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e6ac13b19fb6427a9ef35a5160033a54 (table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]) successfully deleted
I20250905 19:52:38.878397 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e5fc2c2ba49444af939f5a975165362b" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.878966 711 leader_election.cc:304] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:38.879401 3282 raft_consensus.cc:2747] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.891007 3299 consensus_queue.cc:1035] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:38.891940 986 leader_election.cc:343] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:38.892930 3268 raft_consensus.cc:491] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.893002 2413 raft_consensus.cc:491] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.893129 2413 raft_consensus.cc:513] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.893683 2413 leader_election.cc:290] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.894295 3268 raft_consensus.cc:513] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:38.895177 3268 leader_election.cc:290] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.896024 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:38.896178 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad956757086b4d728698679c63c9b7ee" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.896859 848 leader_election.cc:343] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.897931 849 leader_election.cc:343] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.898933 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aff2acc3f8245e1bca41f971f8e74ce" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.899099 907 raft_consensus.cc:2466] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.895992 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad956757086b4d728698679c63c9b7ee" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.900097 849 leader_election.cc:343] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.900250 849 leader_election.cc:304] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.900799 1169 tablet_bootstrap.cc:492] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.901008 1169 ts_tablet_manager.cc:1397] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.106s user 0.001s sys 0.001s
I20250905 19:52:38.901546 2422 raft_consensus.cc:2747] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.901968 1169 raft_consensus.cc:357] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.892167 986 leader_election.cc:304] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.902148 1169 raft_consensus.cc:383] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.902201 1169 raft_consensus.cc:738] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:38.902246 986 raft_consensus.cc:2684] T e6ac13b19fb6427a9ef35a5160033a54 P 4eb3af79b01d4835bf6bf7b26bf016f5: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:38.902388 1169 consensus_queue.cc:260] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.903261 2496 consensus_queue.cc:1035] T ca646f7c162e41469f283657e72823b4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:38.903806 2413 raft_consensus.cc:491] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.903934 2413 raft_consensus.cc:513] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.904085 1169 ts_tablet_manager.cc:1428] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20250905 19:52:38.904336 1169 tablet_bootstrap.cc:492] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.904595 2413 leader_election.cc:290] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.904966 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:38.905401 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.905406 848 leader_election.cc:343] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:38.905911 849 leader_election.cc:343] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.906044 849 leader_election.cc:304] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.906723 3268 raft_consensus.cc:2747] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.907521 1171 tablet_bootstrap.cc:492] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:38.907644 1171 ts_tablet_manager.cc:1397] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.051s user 0.007s sys 0.005s
I20250905 19:52:38.908105 1169 tablet_bootstrap.cc:654] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.908409 1171 raft_consensus.cc:357] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.908572 1171 raft_consensus.cc:383] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.908622 1171 raft_consensus.cc:738] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.908793 1171 consensus_queue.cc:260] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.908923 2998 raft_consensus.cc:491] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.909037 2998 raft_consensus.cc:513] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.909224 1171 ts_tablet_manager.cc:1428] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:38.910276 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc5428f20ca14fdf88a3a52f7b4d1e19" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.910786 984 leader_election.cc:343] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.910933 2998 leader_election.cc:290] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.911262 3253 ts_tablet_manager.cc:1918] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:38.911432 3253 log.cc:1199] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9477afb5b47c46d4b24128a6c4bdabc6
I20250905 19:52:38.911906 3253 ts_tablet_manager.cc:1939] T 9477afb5b47c46d4b24128a6c4bdabc6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:38.912892 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.913427 848 leader_election.cc:343] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.913537 848 leader_election.cc:304] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.913980 3155 raft_consensus.cc:2747] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.915709 1171 tablet_bootstrap.cc:492] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:38.917694 1171 tablet_bootstrap.cc:654] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.923842 3155 raft_consensus.cc:491] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:38.924939 3155 raft_consensus.cc:513] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.925925 3155 leader_election.cc:290] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:38.926477 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:38.926980 849 leader_election.cc:343] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.931151 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:38.932019 848 leader_election.cc:343] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.932152 848 leader_election.cc:304] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.932591 3155 raft_consensus.cc:2747] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:38.941787 1169 tablet_bootstrap.cc:492] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:38.941998 1169 ts_tablet_manager.cc:1397] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.002s
I20250905 19:52:38.942765 1169 raft_consensus.cc:357] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.942997 1169 raft_consensus.cc:383] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.943087 1169 raft_consensus.cc:738] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.943349 1169 consensus_queue.cc:260] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.943943 1169 ts_tablet_manager.cc:1428] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:38.944432 1169 tablet_bootstrap.cc:492] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:38.946019 1169 tablet_bootstrap.cc:654] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.961006 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc5428f20ca14fdf88a3a52f7b4d1e19" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:38.961261 904 raft_consensus.cc:2466] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:38.964114 986 leader_election.cc:304] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:38.964708 2931 raft_consensus.cc:2802] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:38.964785 2931 raft_consensus.cc:491] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:38.964838 2931 raft_consensus.cc:3058] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:38.967553 2931 raft_consensus.cc:513] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.968958 2931 leader_election.cc:290] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:38.970386 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aff2acc3f8245e1bca41f971f8e74ce" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:38.971580 984 leader_election.cc:343] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:38.988813 1170 tablet_bootstrap.cc:492] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:38.988972 1170 ts_tablet_manager.cc:1397] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.182s user 0.015s sys 0.028s
I20250905 19:52:38.989697 1170 raft_consensus.cc:357] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.989874 1170 raft_consensus.cc:383] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:38.989924 1170 raft_consensus.cc:738] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:38.990151 1170 consensus_queue.cc:260] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:38.990530 1170 ts_tablet_manager.cc:1428] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250905 19:52:38.990774 1170 tablet_bootstrap.cc:492] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:38.992524 1170 tablet_bootstrap.cc:654] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:38.999157 986 leader_election.cc:304] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.001890 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5aff2acc3f8245e1bca41f971f8e74ce" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.002085 908 raft_consensus.cc:3058] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.003494 3151 raft_consensus.cc:2802] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.003655 3151 raft_consensus.cc:491] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.003778 3151 raft_consensus.cc:3058] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.004850 908 raft_consensus.cc:2466] T 5aff2acc3f8245e1bca41f971f8e74ce P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.005810 986 leader_election.cc:304] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.006407 2931 raft_consensus.cc:2802] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.006484 3151 raft_consensus.cc:513] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.006574 2931 raft_consensus.cc:695] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.006881 2931 consensus_queue.cc:237] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.012152 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc5428f20ca14fdf88a3a52f7b4d1e19" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:39.012753 984 leader_election.cc:343] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.013178 2998 raft_consensus.cc:491] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.013341 2998 raft_consensus.cc:513] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.013958 2998 leader_election.cc:290] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.007055 3151 leader_election.cc:290] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.016026 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.016461 984 leader_election.cc:343] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.016327 1170 tablet_bootstrap.cc:492] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.017983 1170 ts_tablet_manager.cc:1397] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.027s user 0.003s sys 0.003s
I20250905 19:52:39.018894 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "dc5428f20ca14fdf88a3a52f7b4d1e19" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.018980 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.019083 909 raft_consensus.cc:3058] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.019157 905 raft_consensus.cc:2466] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.019222 3155 raft_consensus.cc:491] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.019312 3155 raft_consensus.cc:513] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:39.019673 1170 raft_consensus.cc:357] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.019827 1170 raft_consensus.cc:383] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.019876 1170 raft_consensus.cc:738] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.020246 986 leader_election.cc:304] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.020303 3155 leader_election.cc:290] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.020314 1170 consensus_queue.cc:260] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.020762 2998 raft_consensus.cc:2802] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.020788 1170 ts_tablet_manager.cc:1428] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:39.020838 2998 raft_consensus.cc:491] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.020895 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff0e7cc7c814627ad8964fa6a3b1576" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.021044 1170 tablet_bootstrap.cc:492] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:39.021430 849 leader_election.cc:343] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.020895 2998 raft_consensus.cc:3058] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.021831 909 raft_consensus.cc:2466] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.022682 1170 tablet_bootstrap.cc:654] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.023310 1169 tablet_bootstrap.cc:492] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.023504 1169 ts_tablet_manager.cc:1397] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.079s user 0.000s sys 0.003s
I20250905 19:52:39.024400 1169 raft_consensus.cc:357] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.024896 1169 raft_consensus.cc:383] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.024991 1169 raft_consensus.cc:738] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.025153 2998 raft_consensus.cc:513] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.016744 770 raft_consensus.cc:3058] T e5fc2c2ba49444af939f5a975165362b P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.026202 3297 consensus_queue.cc:1035] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20250905 19:52:39.027316 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:39.027854 984 leader_election.cc:343] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.028201 1170 tablet_bootstrap.cc:492] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.028322 1170 ts_tablet_manager.cc:1397] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.001s
I20250905 19:52:39.028934 986 leader_election.cc:304] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.030326 3151 raft_consensus.cc:2802] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.030491 3151 raft_consensus.cc:695] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.030789 3151 consensus_queue.cc:237] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.025241 1169 consensus_queue.cc:260] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.033195 1170 raft_consensus.cc:357] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.033376 1170 raft_consensus.cc:383] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.033430 1170 raft_consensus.cc:738] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.033650 1170 consensus_queue.cc:260] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.034127 1170 ts_tablet_manager.cc:1428] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:39.034404 1170 tablet_bootstrap.cc:492] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.036540 1169 ts_tablet_manager.cc:1428] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.013s user 0.001s sys 0.000s
I20250905 19:52:39.036871 1169 tablet_bootstrap.cc:492] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.038097 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff0e7cc7c814627ad8964fa6a3b1576" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.038650 848 leader_election.cc:343] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.038762 848 leader_election.cc:304] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.039987 3155 raft_consensus.cc:2747] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.045022 1170 tablet_bootstrap.cc:654] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.049156 1170 tablet_bootstrap.cc:492] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.049291 1170 ts_tablet_manager.cc:1397] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.003s
I20250905 19:52:39.049975 2998 leader_election.cc:290] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.050014 1170 raft_consensus.cc:357] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.050187 1170 raft_consensus.cc:383] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.050267 769 raft_consensus.cc:3058] T ca646f7c162e41469f283657e72823b4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.050256 1170 raft_consensus.cc:738] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.051955 1170 consensus_queue.cc:260] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.052423 1170 ts_tablet_manager.cc:1428] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20250905 19:52:39.052675 1170 tablet_bootstrap.cc:492] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.053498 764 raft_consensus.cc:3058] T d495b6ba92e84b588b4780c2896195c0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.054275 1170 tablet_bootstrap.cc:654] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.069912 3155 raft_consensus.cc:491] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.070171 3155 raft_consensus.cc:513] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.071300 3155 leader_election.cc:290] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.071807 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:39.072440 849 leader_election.cc:343] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.075686 1171 tablet_bootstrap.cc:492] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:39.075862 1171 ts_tablet_manager.cc:1397] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.160s user 0.015s sys 0.030s
I20250905 19:52:39.076660 1171 raft_consensus.cc:357] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.076901 1171 raft_consensus.cc:383] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.076992 1171 raft_consensus.cc:738] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.077262 1171 consensus_queue.cc:260] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.077874 1171 ts_tablet_manager.cc:1428] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:39.078230 1171 tablet_bootstrap.cc:492] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:39.086395 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.087450 848 leader_election.cc:343] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.087575 848 leader_election.cc:304] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.088096 2422 raft_consensus.cc:2747] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:39.088788 984 consensus_peers.cc:489] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:39.097101 3350 raft_consensus.cc:491] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:39.100348 3350 raft_consensus.cc:513] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:39.101123 1170 tablet_bootstrap.cc:492] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.101325 1170 ts_tablet_manager.cc:1397] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.006s
I20250905 19:52:39.101696 2452 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" candidate_term: 4 candidate_status { last_received { term: 3 index: 689 } } ignore_live_leader: false dest_uuid: "c401c9f3020b4ae5931c566c6bcb1711" is_pre_election: true
I20250905 19:52:39.102190 1170 raft_consensus.cc:357] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.102416 1170 raft_consensus.cc:383] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.102520 1170 raft_consensus.cc:738] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.102697 3350 leader_election.cc:290] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c401c9f3020b4ae5931c566c6bcb1711 (127.0.104.62:38687), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:39.102802 1170 consensus_queue.cc:260] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.103309 1170 ts_tablet_manager.cc:1428] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:39.103632 1170 tablet_bootstrap.cc:492] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.104116 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.104285 906 raft_consensus.cc:3058] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.105041 2931 consensus_queue.cc:1035] T 91396422d0bb468395db04aa80262882 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.105261 1170 tablet_bootstrap.cc:654] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.113540 3354 raft_consensus.cc:491] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:39.116469 3354 raft_consensus.cc:513] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:39.116837 1170 tablet_bootstrap.cc:492] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.117013 1170 ts_tablet_manager.cc:1397] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.002s
I20250905 19:52:39.120909 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 4 candidate_status { last_received { term: 3 index: 689 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" is_pre_election: true
I20250905 19:52:39.121129 536 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c401c9f3020b4ae5931c566c6bcb1711 in term 3.
I20250905 19:52:39.121784 2285 leader_election.cc:304] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711, c6c0f2b9d8cd457a9c3448034c3d5c91; no voters:
I20250905 19:52:39.129045 3354 leader_election.cc:290] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:39.129681 3354 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20250905 19:52:39.130402 3354 raft_consensus.cc:491] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 3 FOLLOWER]: Starting leader election (detected failure of leader e38333af29db4bf6af41ee5a25d6701e)
I20250905 19:52:39.130760 3354 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:52:39.133589 1171 tablet_bootstrap.cc:654] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.134341 1170 raft_consensus.cc:357] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.134529 1170 raft_consensus.cc:383] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.134577 1170 raft_consensus.cc:738] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.134763 1170 consensus_queue.cc:260] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.135289 1170 ts_tablet_manager.cc:1428] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.018s user 0.000s sys 0.000s
I20250905 19:52:39.135576 1170 tablet_bootstrap.cc:492] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.136382 906 raft_consensus.cc:2466] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.137199 986 leader_election.cc:304] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.137552 2496 raft_consensus.cc:2802] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.137650 2496 raft_consensus.cc:695] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.137799 1170 tablet_bootstrap.cc:654] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.137967 3346 raft_consensus.cc:491] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.138149 3346 raft_consensus.cc:513] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.138499 2496 consensus_queue.cc:237] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.140033 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "862214a31059407baefef8a390aaea69" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.140476 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "862214a31059407baefef8a390aaea69" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.141268 712 leader_election.cc:304] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.141534 1171 tablet_bootstrap.cc:492] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:39.141517 3282 raft_consensus.cc:2747] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.141652 1171 ts_tablet_manager.cc:1397] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20250905 19:52:39.142283 3346 leader_election.cc:290] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.142411 1171 raft_consensus.cc:357] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.142561 1171 raft_consensus.cc:383] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.142617 1171 raft_consensus.cc:738] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.142804 1171 consensus_queue.cc:260] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.151201 1169 tablet_bootstrap.cc:654] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.152590 1171 ts_tablet_manager.cc:1428] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.011s user 0.001s sys 0.001s
I20250905 19:52:39.153602 1171 tablet_bootstrap.cc:492] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:39.167169 3354 raft_consensus.cc:513] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:39.171749 1171 tablet_bootstrap.cc:654] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.176067 536 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c401c9f3020b4ae5931c566c6bcb1711" candidate_term: 4 candidate_status { last_received { term: 3 index: 689 } } ignore_live_leader: false dest_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91"
I20250905 19:52:39.176296 536 raft_consensus.cc:3058] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 3 FOLLOWER]: Advancing to term 4
I20250905 19:52:39.179054 536 raft_consensus.cc:2466] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c401c9f3020b4ae5931c566c6bcb1711 in term 4.
I20250905 19:52:39.179797 2285 leader_election.cc:304] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c401c9f3020b4ae5931c566c6bcb1711, c6c0f2b9d8cd457a9c3448034c3d5c91; no voters:
I20250905 19:52:39.180217 3356 raft_consensus.cc:2802] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 4 FOLLOWER]: Leader election won for term 4
I20250905 19:52:39.180794 3354 leader_election.cc:290] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 4 election: Requested vote from peers c6c0f2b9d8cd457a9c3448034c3d5c91 (127.0.104.61:41317), e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395)
I20250905 19:52:39.184028 769 raft_consensus.cc:3058] T 862214a31059407baefef8a390aaea69 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.184857 1170 tablet_bootstrap.cc:492] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.184975 1170 ts_tablet_manager.cc:1397] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.049s user 0.002s sys 0.006s
I20250905 19:52:39.185148 2422 raft_consensus.cc:491] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.185292 2422 raft_consensus.cc:513] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.185660 1170 raft_consensus.cc:357] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.185799 1170 raft_consensus.cc:383] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.185838 1170 raft_consensus.cc:738] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.185822 2422 leader_election.cc:290] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.186043 1170 consensus_queue.cc:260] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.187004 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:39.187808 849 leader_election.cc:343] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.188172 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.188692 848 leader_election.cc:343] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.188836 848 leader_election.cc:304] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.189122 2422 raft_consensus.cc:2747] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.195096 821 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:39.202220 1171 tablet_bootstrap.cc:492] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:39.202387 1171 ts_tablet_manager.cc:1397] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.049s user 0.000s sys 0.005s
I20250905 19:52:39.208695 1171 raft_consensus.cc:357] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.209061 1171 raft_consensus.cc:383] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.209117 1171 raft_consensus.cc:738] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.209017 1170 ts_tablet_manager.cc:1428] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.024s user 0.001s sys 0.001s
I20250905 19:52:39.209321 1171 consensus_queue.cc:260] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.209599 1170 tablet_bootstrap.cc:492] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.209789 1171 ts_tablet_manager.cc:1428] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20250905 19:52:39.210042 1171 tablet_bootstrap.cc:492] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:39.211309 1170 tablet_bootstrap.cc:654] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.211530 1171 tablet_bootstrap.cc:654] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.218073 2422 raft_consensus.cc:491] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.218276 2422 raft_consensus.cc:513] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.219034 2422 leader_election.cc:290] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.219272 2496 raft_consensus.cc:491] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.219404 2496 raft_consensus.cc:513] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.219990 2496 leader_election.cc:290] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.221159 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.221339 907 raft_consensus.cc:2466] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.221959 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.222581 3346 raft_consensus.cc:491] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.222748 986 leader_election.cc:304] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:39.222729 3346 raft_consensus.cc:513] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:39.223121 849 leader_election.cc:343] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.223414 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.223524 3346 leader_election.cc:290] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:39.224005 984 leader_election.cc:343] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.224732 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.225436 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d1a11777cf2d4ea9a5cdf792f73aff76" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.225656 907 raft_consensus.cc:2391] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:39.226325 711 leader_election.cc:304] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.226639 3346 raft_consensus.cc:3058] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.226907 2496 raft_consensus.cc:2802] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.227001 2496 raft_consensus.cc:491] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.227077 2496 raft_consensus.cc:3058] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.230223 3346 raft_consensus.cc:2747] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.236071 3356 raft_consensus.cc:695] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [term 4 LEADER]: Becoming Leader. State: Replica: c401c9f3020b4ae5931c566c6bcb1711, State: Running, Role: LEADER
I20250905 19:52:39.236372 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.236862 3356 consensus_queue.cc:237] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 689, Committed index: 689, Last appended: 3.689, Last appended by leader: 689, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } }
I20250905 19:52:39.238907 1169 tablet_bootstrap.cc:492] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.239037 1169 ts_tablet_manager.cc:1397] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.202s user 0.012s sys 0.018s
I20250905 19:52:39.239745 1169 raft_consensus.cc:357] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.239938 1169 raft_consensus.cc:383] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.240015 1169 raft_consensus.cc:738] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.240201 1169 consensus_queue.cc:260] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.240691 1170 tablet_bootstrap.cc:492] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.240813 1170 ts_tablet_manager.cc:1397] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.031s user 0.002s sys 0.000s
I20250905 19:52:39.241039 2496 raft_consensus.cc:513] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.241447 1170 raft_consensus.cc:357] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.241602 1170 raft_consensus.cc:383] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.241652 1170 raft_consensus.cc:738] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.241815 1170 consensus_queue.cc:260] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.241935 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:39.242408 984 leader_election.cc:343] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.244418 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.244614 907 raft_consensus.cc:3058] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.244632 1170 ts_tablet_manager.cc:1428] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:39.244913 1170 tablet_bootstrap.cc:492] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.245277 3354 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c401c9f3020b4ae5931c566c6bcb1711. Latest consensus state: current_term: 4 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:39.245496 3354 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:39.246380 1170 tablet_bootstrap.cc:654] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.247251 907 raft_consensus.cc:2466] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.247769 986 leader_election.cc:304] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:39.248310 848 leader_election.cc:343] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.248418 848 leader_election.cc:304] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.248843 2422 raft_consensus.cc:2747] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.251327 2496 leader_election.cc:290] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.251680 2496 raft_consensus.cc:2802] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.251933 2496 raft_consensus.cc:695] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.252249 2496 consensus_queue.cc:237] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.265671 3366 catalog_manager.cc:1477] Loading table and tablet metadata into memory...
I20250905 19:52:39.266260 1169 ts_tablet_manager.cc:1428] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250905 19:52:39.266675 1169 tablet_bootstrap.cc:492] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.268715 1169 tablet_bootstrap.cc:654] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.274402 1171 tablet_bootstrap.cc:492] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:39.274569 1171 ts_tablet_manager.cc:1397] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.065s user 0.007s sys 0.007s
I20250905 19:52:39.275287 1171 raft_consensus.cc:357] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.275466 1171 raft_consensus.cc:383] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.275514 1171 raft_consensus.cc:738] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.275791 1171 consensus_queue.cc:260] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.276371 1171 ts_tablet_manager.cc:1428] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:39.278484 1169 tablet_bootstrap.cc:492] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.278623 1169 ts_tablet_manager.cc:1397] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250905 19:52:39.281168 1171 tablet_bootstrap.cc:492] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:39.283766 1171 tablet_bootstrap.cc:654] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.295045 2422 raft_consensus.cc:491] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.295229 2422 raft_consensus.cc:513] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.295835 2422 leader_election.cc:290] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.296254 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.296604 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
W20250905 19:52:39.296777 848 leader_election.cc:343] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:39.297163 849 leader_election.cc:343] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.297256 849 leader_election.cc:304] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.297672 3268 raft_consensus.cc:2747] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:39.314591 984 consensus_peers.cc:489] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:39.314824 1170 tablet_bootstrap.cc:492] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.314962 1170 ts_tablet_manager.cc:1397] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.070s user 0.002s sys 0.002s
I20250905 19:52:39.315256 1169 raft_consensus.cc:357] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.315445 1169 raft_consensus.cc:383] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.315497 1169 raft_consensus.cc:738] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.315627 1170 raft_consensus.cc:357] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.315804 1170 raft_consensus.cc:383] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.315724 1169 consensus_queue.cc:260] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.315862 1170 raft_consensus.cc:738] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.316078 1170 consensus_queue.cc:260] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.316561 1170 ts_tablet_manager.cc:1428] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:39.316865 1170 tablet_bootstrap.cc:492] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.318428 1170 tablet_bootstrap.cc:654] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.319692 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2]
I20250905 19:52:39.320418 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a]
I20250905 19:52:39.326808 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]
I20250905 19:52:39.327425 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b]
I20250905 19:52:39.333861 3268 raft_consensus.cc:491] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.334055 3268 raft_consensus.cc:513] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.334597 1169 ts_tablet_manager.cc:1428] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.056s user 0.002s sys 0.000s
I20250905 19:52:39.334647 3268 leader_election.cc:290] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.334952 1169 tablet_bootstrap.cc:492] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.335574 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.336062 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.336710 1169 tablet_bootstrap.cc:654] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.336683 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_d090d4e81b7c45e49c707734cfeae659 [id=02eadd9157e44aaf861adbfd95b2fc9c]
W20250905 19:52:39.336704 849 leader_election.cc:343] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:39.337080 848 leader_election.cc:343] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.337232 848 leader_election.cc:304] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.337653 3268 raft_consensus.cc:2747] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.340947 1170 tablet_bootstrap.cc:492] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.341100 1170 ts_tablet_manager.cc:1397] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.024s user 0.002s sys 0.004s
I20250905 19:52:39.341599 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_cfe02f65c42d492b9ebd6d9891ff4b34 [id=03ed72cf63184003be5e02061d130ce3]
I20250905 19:52:39.341804 1170 raft_consensus.cc:357] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.341974 1170 raft_consensus.cc:383] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.342026 1170 raft_consensus.cc:738] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.342271 1170 consensus_queue.cc:260] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.349630 1171 tablet_bootstrap.cc:492] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:39.349630 1169 tablet_bootstrap.cc:492] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.349766 1169 ts_tablet_manager.cc:1397] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.000s
I20250905 19:52:39.349766 1171 ts_tablet_manager.cc:1397] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.069s user 0.010s sys 0.004s
I20250905 19:52:39.350487 1169 raft_consensus.cc:357] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.350486 1171 raft_consensus.cc:357] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.350643 1171 raft_consensus.cc:383] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.350639 1169 raft_consensus.cc:383] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.350692 1171 raft_consensus.cc:738] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.350692 1169 raft_consensus.cc:738] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.350880 1169 consensus_queue.cc:260] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.350941 1171 consensus_queue.cc:260] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.352332 1170 ts_tablet_manager.cc:1428] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:39.352658 1170 tablet_bootstrap.cc:492] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.353222 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906]
I20250905 19:52:39.354130 1170 tablet_bootstrap.cc:654] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.354728 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d]
I20250905 19:52:39.355404 1171 ts_tablet_manager.cc:1428] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:39.358894 3346 raft_consensus.cc:491] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.359018 3346 raft_consensus.cc:513] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.359632 3346 leader_election.cc:290] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.360251 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.360888 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "b1f46261822442e0a961d2d63bdb35c8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.361073 907 raft_consensus.cc:2391] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:39.361439 712 leader_election.cc:304] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.361816 3346 raft_consensus.cc:3058] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.364511 3346 raft_consensus.cc:2747] T b1f46261822442e0a961d2d63bdb35c8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.367379 1169 ts_tablet_manager.cc:1428] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20250905 19:52:39.367671 1169 tablet_bootstrap.cc:492] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.369385 1169 tablet_bootstrap.cc:654] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.370875 2496 raft_consensus.cc:491] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.370990 2496 raft_consensus.cc:513] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.371564 2496 leader_election.cc:290] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.372061 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c69a41cee5ff4f8f9bccaa513c096118" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.372241 907 raft_consensus.cc:2466] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.372732 986 leader_election.cc:304] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:39.373464 2496 raft_consensus.cc:2802] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.373543 2496 raft_consensus.cc:491] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.373504 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c69a41cee5ff4f8f9bccaa513c096118" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.373601 2496 raft_consensus.cc:3058] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.373689 770 raft_consensus.cc:2466] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.376508 2496 raft_consensus.cc:513] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.377374 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c69a41cee5ff4f8f9bccaa513c096118" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:39.377537 770 raft_consensus.cc:3058] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.378966 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]
I20250905 19:52:39.380944 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c69a41cee5ff4f8f9bccaa513c096118" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.381124 907 raft_consensus.cc:3058] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.383863 907 raft_consensus.cc:2466] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.384263 2496 leader_election.cc:290] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.384655 770 raft_consensus.cc:2466] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.385092 984 leader_election.cc:304] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:39.385881 2496 raft_consensus.cc:2802] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.386049 2496 raft_consensus.cc:695] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.386354 2496 consensus_queue.cc:237] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.391527 769 raft_consensus.cc:3058] T 923ad8e40a07409e8f6a0152a0b2ef75 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.379812 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]
I20250905 19:52:39.392835 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_44734742f0c948af9cce54466ff759c6 [id=11511883d6b046c7b155ff9687eb2c76]
I20250905 19:52:39.393683 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3]
I20250905 19:52:39.395264 770 raft_consensus.cc:3058] T 180437d1534946619384307ddcb7c937 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.395336 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba]
I20250905 19:52:39.395884 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c]
I20250905 19:52:39.396396 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_85e1b8aa7bd340459631f0b02ae94b9d [id=151cf052220f4749be3757d20864c425]
I20250905 19:52:39.399938 3274 consensus_queue.cc:1035] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.400523 3274 consensus_queue.cc:1035] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.397120 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]
I20250905 19:52:39.405194 3151 consensus_queue.cc:1035] T 923ad8e40a07409e8f6a0152a0b2ef75 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.408571 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]
I20250905 19:52:39.410120 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]
I20250905 19:52:39.418416 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b24d9f16949145cda6b4f82d78993fa0 [id=2242efd05b0e4fae989ce99e9431aa4b]
I20250905 19:52:39.421262 1170 tablet_bootstrap.cc:492] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.421418 1170 ts_tablet_manager.cc:1397] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.000s
I20250905 19:52:39.423830 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6]
I20250905 19:52:39.426105 1170 raft_consensus.cc:357] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.426365 1170 raft_consensus.cc:383] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.426430 1170 raft_consensus.cc:738] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.426641 1170 consensus_queue.cc:260] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.427165 1170 ts_tablet_manager.cc:1428] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20250905 19:52:39.427472 1170 tablet_bootstrap.cc:492] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.427701 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]
I20250905 19:52:39.428917 1170 tablet_bootstrap.cc:654] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.430481 1169 tablet_bootstrap.cc:492] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.430619 1169 ts_tablet_manager.cc:1397] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.002s
I20250905 19:52:39.431344 1169 raft_consensus.cc:357] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.431511 1169 raft_consensus.cc:383] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.431562 1169 raft_consensus.cc:738] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.431747 1169 consensus_queue.cc:260] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.432266 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f]
I20250905 19:52:39.432487 1169 ts_tablet_manager.cc:1428] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:39.432756 1169 tablet_bootstrap.cc:492] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.434764 1169 tablet_bootstrap.cc:654] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.435304 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a]
I20250905 19:52:39.440076 3297 consensus_queue.cc:1035] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.440815 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967]
I20250905 19:52:39.441546 2413 raft_consensus.cc:491] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.441677 2413 raft_consensus.cc:513] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.442332 2413 leader_election.cc:290] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.442792 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5]
I20250905 19:52:39.443328 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_c30c09bc294d465da486648091481ed1 [id=3118b283e4194c5fa5c5a7e270d14e49]
I20250905 19:52:39.443552 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2b5ce70b784c13aa64053b2d1fe541" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.443848 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e]
W20250905 19:52:39.444087 848 leader_election.cc:343] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.444226 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2b5ce70b784c13aa64053b2d1fe541" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.444677 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c]
I20250905 19:52:39.445086 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6]
W20250905 19:52:39.449527 849 leader_election.cc:343] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.449669 849 leader_election.cc:304] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.450340 3155 raft_consensus.cc:2747] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.447932 1098 heartbeater.cc:499] Master 127.0.104.60:34395 was elected leader, sending a full tablet report...
I20250905 19:52:39.475472 2931 raft_consensus.cc:491] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.475639 2931 raft_consensus.cc:513] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.477011 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.477496 984 leader_election.cc:343] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.480159 3297 consensus_queue.cc:1035] T 180437d1534946619384307ddcb7c937 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.482506 2931 leader_election.cc:290] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.483323 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.483566 907 raft_consensus.cc:2466] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.484557 986 leader_election.cc:304] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.486097 2413 raft_consensus.cc:491] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.486209 2413 raft_consensus.cc:513] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:39.486690 2496 raft_consensus.cc:2802] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.486725 2413 leader_election.cc:290] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.486770 2496 raft_consensus.cc:491] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.486830 2496 raft_consensus.cc:3058] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.487036 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92]
I20250905 19:52:39.487636 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.487954 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.488534 848 leader_election.cc:343] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:39.488767 849 leader_election.cc:343] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.488863 849 leader_election.cc:304] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.489289 2413 raft_consensus.cc:2747] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.490953 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]
I20250905 19:52:39.493568 3282 raft_consensus.cc:491] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.493736 3282 raft_consensus.cc:513] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.494354 3282 leader_election.cc:290] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.494721 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.495513 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "d59da39e7bcf497a93fb9174328c1814" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.496094 711 leader_election.cc:304] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.496511 3282 raft_consensus.cc:2747] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.505371 1170 tablet_bootstrap.cc:492] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.505519 1170 ts_tablet_manager.cc:1397] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.078s user 0.011s sys 0.003s
I20250905 19:52:39.506193 1170 raft_consensus.cc:357] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.506358 1170 raft_consensus.cc:383] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.506405 1170 raft_consensus.cc:738] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.506631 1170 consensus_queue.cc:260] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.507082 1170 ts_tablet_manager.cc:1428] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.001s user 0.001s sys 0.001s
I20250905 19:52:39.507354 1170 tablet_bootstrap.cc:492] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.508916 1170 tablet_bootstrap.cc:654] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.517786 1169 tablet_bootstrap.cc:492] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.517928 1169 ts_tablet_manager.cc:1397] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.085s user 0.006s sys 0.012s
I20250905 19:52:39.518635 1169 raft_consensus.cc:357] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.518791 1169 raft_consensus.cc:383] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.518838 1169 raft_consensus.cc:738] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.519040 1169 consensus_queue.cc:260] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.521112 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1]
I20250905 19:52:39.523135 1169 ts_tablet_manager.cc:1428] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20250905 19:52:39.523422 1169 tablet_bootstrap.cc:492] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.525154 1169 tablet_bootstrap.cc:654] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.528524 1170 tablet_bootstrap.cc:492] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.528674 1170 ts_tablet_manager.cc:1397] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20250905 19:52:39.529557 1170 raft_consensus.cc:357] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.529745 1170 raft_consensus.cc:383] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.529803 1170 raft_consensus.cc:738] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.530006 1170 consensus_queue.cc:260] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.530522 1170 ts_tablet_manager.cc:1428] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:39.530820 1170 tablet_bootstrap.cc:492] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.533043 1170 tablet_bootstrap.cc:654] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.534035 3282 raft_consensus.cc:491] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.534164 3282 raft_consensus.cc:513] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.534735 3282 leader_election.cc:290] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:39.535821 712 leader_election.cc:343] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 14d5eb8bec074d2c8766fe7b159031e5
W20250905 19:52:39.535991 711 leader_election.cc:343] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 14d5eb8bec074d2c8766fe7b159031e5
I20250905 19:52:39.536111 711 leader_election.cc:304] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.536434 3282 raft_consensus.cc:2747] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.539742 3282 raft_consensus.cc:491] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.539873 3282 raft_consensus.cc:513] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.540575 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646]
I20250905 19:52:39.540635 3282 leader_election.cc:290] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.541822 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]
I20250905 19:52:39.542382 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]
I20250905 19:52:39.542551 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be1315824fd749049a62af3e6a113b9a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.543558 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d]
I20250905 19:52:39.547748 764 raft_consensus.cc:3058] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.547816 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "be1315824fd749049a62af3e6a113b9a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.548143 2496 raft_consensus.cc:513] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.549158 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.549211 711 leader_election.cc:304] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.549337 907 raft_consensus.cc:3058] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.552152 907 raft_consensus.cc:2466] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.552207 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:39.552647 986 leader_election.cc:304] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:39.552798 984 leader_election.cc:343] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.553071 2931 raft_consensus.cc:2802] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.553180 2931 raft_consensus.cc:695] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.553433 2931 consensus_queue.cc:237] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.557912 1170 tablet_bootstrap.cc:492] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.558066 1170 ts_tablet_manager.cc:1397] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.027s user 0.000s sys 0.007s
I20250905 19:52:39.558779 1170 raft_consensus.cc:357] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.558954 1170 raft_consensus.cc:383] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.559023 1170 raft_consensus.cc:738] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.559470 2496 leader_election.cc:290] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.559895 1170 consensus_queue.cc:260] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.560433 1170 ts_tablet_manager.cc:1428] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:39.560690 1170 tablet_bootstrap.cc:492] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.562213 1170 tablet_bootstrap.cc:654] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.563616 3282 raft_consensus.cc:2747] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.563665 1169 tablet_bootstrap.cc:492] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.563772 1169 ts_tablet_manager.cc:1397] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.004s
I20250905 19:52:39.564528 1169 raft_consensus.cc:357] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.564690 1169 raft_consensus.cc:383] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.564755 1169 raft_consensus.cc:738] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.565387 1169 consensus_queue.cc:260] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.566040 1169 ts_tablet_manager.cc:1428] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:39.566294 1169 tablet_bootstrap.cc:492] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.568120 1169 tablet_bootstrap.cc:654] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.569936 1170 tablet_bootstrap.cc:492] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.570057 1170 ts_tablet_manager.cc:1397] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20250905 19:52:39.570696 1170 raft_consensus.cc:357] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.570859 1170 raft_consensus.cc:383] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.570905 1170 raft_consensus.cc:738] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.571107 1170 consensus_queue.cc:260] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.571764 1170 ts_tablet_manager.cc:1428] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:39.572050 1170 tablet_bootstrap.cc:492] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.573596 1170 tablet_bootstrap.cc:654] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.582544 3297 consensus_queue.cc:1035] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.592085 764 raft_consensus.cc:3058] T 65368306f86845008d4d486023e1fc29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.593595 1169 tablet_bootstrap.cc:492] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.593716 1169 ts_tablet_manager.cc:1397] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.002s
I20250905 19:52:39.594401 1169 raft_consensus.cc:357] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.594586 1169 raft_consensus.cc:383] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.594636 1169 raft_consensus.cc:738] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.594831 1169 consensus_queue.cc:260] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.595378 1169 ts_tablet_manager.cc:1428] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:39.595661 1169 tablet_bootstrap.cc:492] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.595662 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]
I20250905 19:52:39.596191 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6e40598f9dbc40ac9860eb5b78130a93 [id=408552212a1f42089f1d4cb87b4a13b5]
I20250905 19:52:39.597208 1169 tablet_bootstrap.cc:654] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.597345 2496 raft_consensus.cc:491] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.598310 2496 raft_consensus.cc:513] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.599392 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a551e641a19f4d3f924fb176c4a25ca7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.599862 2496 leader_election.cc:290] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.599881 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a551e641a19f4d3f924fb176c4a25ca7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:39.599946 984 leader_election.cc:343] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.600092 906 raft_consensus.cc:2466] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.600533 986 leader_election.cc:304] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.600836 2931 raft_consensus.cc:2802] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.600932 1170 tablet_bootstrap.cc:492] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.600935 2931 raft_consensus.cc:491] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.601079 2931 raft_consensus.cc:3058] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.601080 1170 ts_tablet_manager.cc:1397] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.000s
I20250905 19:52:39.601930 1170 raft_consensus.cc:357] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.602094 1170 raft_consensus.cc:383] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.602141 1170 raft_consensus.cc:738] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.603103 1170 consensus_queue.cc:260] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.608732 769 raft_consensus.cc:3058] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.611384 1170 ts_tablet_manager.cc:1428] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.010s user 0.002s sys 0.001s
I20250905 19:52:39.611667 1170 tablet_bootstrap.cc:492] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.612506 3282 raft_consensus.cc:491] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.612615 3282 raft_consensus.cc:513] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.613101 2413 raft_consensus.cc:491] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.613163 3282 leader_election.cc:290] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.613221 2413 raft_consensus.cc:513] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.613781 2413 leader_election.cc:290] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.614230 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad]
I20250905 19:52:39.614619 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10fef848b9b746c5a0de6a8c3ca74f7a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.614719 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_0d6fe65405f543d7a70643f94bf22da5 [id=46715848277847218d557157e4dfc77b]
I20250905 19:52:39.614976 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10fef848b9b746c5a0de6a8c3ca74f7a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.615108 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_c1b824c37b3b4413b20c5b2172059490 [id=47bee8087f0943199f82be1345be78fd]
W20250905 19:52:39.615275 848 leader_election.cc:343] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:39.615334 712 leader_election.cc:343] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 6ebab2e3698f4f60a770bde9cc019600
I20250905 19:52:39.615509 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e]
W20250905 19:52:39.615694 849 leader_election.cc:343] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.615797 849 leader_election.cc:304] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.616178 2422 raft_consensus.cc:2747] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:39.616979 711 leader_election.cc:343] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 6ebab2e3698f4f60a770bde9cc019600
I20250905 19:52:39.617107 711 leader_election.cc:304] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.617473 3282 raft_consensus.cc:2747] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.618665 1170 tablet_bootstrap.cc:654] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.623040 2931 raft_consensus.cc:513] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.623698 2931 leader_election.cc:290] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.624080 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_3e897c08965d4265b2a5937b15f5c715 [id=48d7bc33c38d44fc977294a75d895a15]
I20250905 19:52:39.624495 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e]
I20250905 19:52:39.625161 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]
I20250905 19:52:39.625607 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c]
I20250905 19:52:39.626530 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a551e641a19f4d3f924fb176c4a25ca7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:39.626685 764 raft_consensus.cc:3058] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.626945 536 raft_consensus.cc:1273] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 4 FOLLOWER]: Refusing update from remote peer c401c9f3020b4ae5931c566c6bcb1711: Log matching property violated. Preceding OpId in replica: term: 3 index: 689. Preceding OpId from leader: term: 4 index: 690. (index mismatch)
I20250905 19:52:39.627234 3299 consensus_queue.cc:1035] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.628547 1169 tablet_bootstrap.cc:492] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.628666 1169 ts_tablet_manager.cc:1397] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20250905 19:52:39.629356 1169 raft_consensus.cc:357] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.629493 1169 raft_consensus.cc:383] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.629534 1169 raft_consensus.cc:738] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.629706 1169 consensus_queue.cc:260] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.630136 1169 ts_tablet_manager.cc:1428] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 19:52:39.630376 1169 tablet_bootstrap.cc:492] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.631916 1169 tablet_bootstrap.cc:654] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.632551 3299 raft_consensus.cc:491] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.632658 3299 raft_consensus.cc:513] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.633212 3299 leader_election.cc:290] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.633688 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74eb2632a6414e52b9ad67f53c182679" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.633865 907 raft_consensus.cc:2466] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.634300 986 leader_election.cc:304] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:39.634502 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74eb2632a6414e52b9ad67f53c182679" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.634598 3299 raft_consensus.cc:2802] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.634694 3299 raft_consensus.cc:491] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.634770 3299 raft_consensus.cc:3058] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:39.635032 984 leader_election.cc:343] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.635373 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "a551e641a19f4d3f924fb176c4a25ca7" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.635514 906 raft_consensus.cc:3058] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.636154 3354 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 690, Last known committed idx: 689, Time since last communication: 0.009s
I20250905 19:52:39.640873 1170 tablet_bootstrap.cc:492] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.641191 1170 ts_tablet_manager.cc:1397] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.000s
I20250905 19:52:39.642040 771 raft_consensus.cc:3058] T e7086c5bfa4644d08e4f55231b85acf4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.642022 1170 raft_consensus.cc:357] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.642633 1170 raft_consensus.cc:383] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.642747 1170 raft_consensus.cc:738] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.643126 1170 consensus_queue.cc:260] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.643792 1170 ts_tablet_manager.cc:1428] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:39.644138 1170 tablet_bootstrap.cc:492] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.645848 1170 tablet_bootstrap.cc:654] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.645814 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f]
I20250905 19:52:39.646452 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422]
I20250905 19:52:39.653379 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]
I20250905 19:52:39.654671 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52]
I20250905 19:52:39.656826 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a]
I20250905 19:52:39.659385 766 raft_consensus.cc:3058] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.660576 2496 consensus_queue.cc:1035] T 65368306f86845008d4d486023e1fc29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.663681 3282 raft_consensus.cc:491] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.663836 3282 raft_consensus.cc:513] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.664664 3282 leader_election.cc:290] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:39.665450 712 leader_election.cc:343] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 421b28d047124222ae9cbc603ec22cee
W20250905 19:52:39.665890 711 leader_election.cc:343] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 421b28d047124222ae9cbc603ec22cee
I20250905 19:52:39.666003 711 leader_election.cc:304] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.666460 3151 raft_consensus.cc:491] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.666507 3282 raft_consensus.cc:2747] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:39.666594 3151 raft_consensus.cc:513] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.667152 3151 leader_election.cc:290] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.667829 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e63b27ff4e434d76a8280f90c22f4082" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.668135 909 raft_consensus.cc:2466] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.668704 986 leader_election.cc:304] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:39.669380 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e63b27ff4e434d76a8280f90c22f4082" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.670128 984 leader_election.cc:343] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.670473 2496 raft_consensus.cc:2802] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.670665 2496 raft_consensus.cc:491] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.670815 2496 raft_consensus.cc:3058] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.673043 3282 raft_consensus.cc:491] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.673213 3282 raft_consensus.cc:513] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.673970 3282 leader_election.cc:290] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.674801 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbd471695e748c9831f9b0f6795aea7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.675060 905 raft_consensus.cc:2466] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:39.676412 712 leader_election.cc:304] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:39.677302 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbd471695e748c9831f9b0f6795aea7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.677512 1048 raft_consensus.cc:2466] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:39.677793 3282 raft_consensus.cc:2802] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.677858 3282 raft_consensus.cc:491] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.677909 3282 raft_consensus.cc:3058] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:39.677996 512 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.60:34395 timed out after 0.574s (SENT) [suppressed 1 similar messages]
I20250905 19:52:39.680171 3375 raft_consensus.cc:491] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.680356 3375 raft_consensus.cc:513] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.682489 3375 leader_election.cc:290] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.682922 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.683501 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5b4f9e12781a46f19b563746a70b59e4" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.685565 905 raft_consensus.cc:2391] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:39.692576 712 leader_election.cc:304] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.693606 3375 raft_consensus.cc:3058] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.695005 3297 consensus_queue.cc:1035] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.708321 764 raft_consensus.cc:2466] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.708698 3299 raft_consensus.cc:513] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.709300 3299 leader_election.cc:290] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.712138 906 raft_consensus.cc:2466] T a551e641a19f4d3f924fb176c4a25ca7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.721145 984 leader_election.cc:304] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:39.721541 3299 raft_consensus.cc:2802] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.721660 3299 raft_consensus.cc:695] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.721879 3299 consensus_queue.cc:237] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.723202 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74eb2632a6414e52b9ad67f53c182679" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:39.724081 984 leader_election.cc:343] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:39.726153 512 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.60:34395, user_credentials={real_user=slave}} blocked reactor thread for 48156.2us
I20250905 19:52:39.725396 3362 sys_catalog.cc:455] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:39.726922 3362 sys_catalog.cc:458] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [sys.catalog]: This master's current role is: LEADER
I20250905 19:52:39.727569 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "74eb2632a6414e52b9ad67f53c182679" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.727739 906 raft_consensus.cc:3058] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.727705 3151 consensus_queue.cc:1035] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.727811 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]
I20250905 19:52:39.728323 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]
I20250905 19:52:39.728945 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_cb111dc4aa3946e8a89529d1cd623c08 [id=5ac15645cb4d4730a4b4464a85299bed]
I20250905 19:52:39.729583 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_c6be55785e1244cb85764306f76c04a2 [id=5ee8596f32f042698dfbe530710fda20]
I20250905 19:52:39.732151 1169 tablet_bootstrap.cc:492] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.732481 1169 ts_tablet_manager.cc:1397] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.102s user 0.021s sys 0.021s
I20250905 19:52:39.733304 1169 raft_consensus.cc:357] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.733546 1169 raft_consensus.cc:383] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.733651 1169 raft_consensus.cc:738] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.733925 1169 consensus_queue.cc:260] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.734520 1169 ts_tablet_manager.cc:1428] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:39.734849 1169 tablet_bootstrap.cc:492] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.736724 1169 tablet_bootstrap.cc:654] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:39.736985 2291 proxy.cc:239] Call had error, refreshing address and retrying: Timed out: RequestConsensusVote RPC to 127.0.104.60:34395 timed out after 0.607s (SENT)
I20250905 19:52:39.740610 3400 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c401c9f3020b4ae5931c566c6bcb1711. Latest consensus state: current_term: 4 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:39.740816 3400 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:39.741259 3412 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "c401c9f3020b4ae5931c566c6bcb1711" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c401c9f3020b4ae5931c566c6bcb1711" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38687 } } peers { permanent_uuid: "c6c0f2b9d8cd457a9c3448034c3d5c91" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 41317 } } peers { permanent_uuid: "e38333af29db4bf6af41ee5a25d6701e" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 34395 } } }
I20250905 19:52:39.741570 3412 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [sys.catalog]: This master's current role is: FOLLOWER
I20250905 19:52:39.748848 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]
I20250905 19:52:39.749461 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]
I20250905 19:52:39.749912 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1]
I20250905 19:52:39.750500 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_94f58de97c5946149d23d635eda4d555 [id=656d24e0a3ad4ac78696df5aec2f279e]
I20250905 19:52:39.752880 2931 raft_consensus.cc:491] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.753067 2931 raft_consensus.cc:513] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.753690 2931 leader_election.cc:290] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.755123 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e8bfe047a6e4cc087eea75244da5b41" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.755648 984 leader_election.cc:343] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.755882 1169 tablet_bootstrap.cc:492] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.756036 1169 ts_tablet_manager.cc:1397] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.021s user 0.006s sys 0.000s
I20250905 19:52:39.756942 1169 raft_consensus.cc:357] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.757128 1169 raft_consensus.cc:383] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.757197 1169 raft_consensus.cc:738] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.757407 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]
I20250905 19:52:39.757405 1169 consensus_queue.cc:260] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.758055 1169 ts_tablet_manager.cc:1428] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:39.758421 1169 tablet_bootstrap.cc:492] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.759763 1169 tablet_bootstrap.cc:654] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.759836 3346 raft_consensus.cc:491] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.759958 3346 raft_consensus.cc:513] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.760561 3346 leader_election.cc:290] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.760739 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e8bfe047a6e4cc087eea75244da5b41" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.760969 909 raft_consensus.cc:2466] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.761202 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.761842 986 leader_election.cc:304] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.762249 3299 raft_consensus.cc:2802] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.762320 3299 raft_consensus.cc:491] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.762373 3299 raft_consensus.cc:3058] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.770962 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f6f29ac5e68d4d8da8f3496fefe695ec" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.771658 711 leader_election.cc:304] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.772125 3346 raft_consensus.cc:2747] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:39.776378 984 consensus_peers.cc:489] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:39.778432 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]
I20250905 19:52:39.781787 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]
I20250905 19:52:39.786736 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499]
I20250905 19:52:39.793045 771 raft_consensus.cc:3058] T f6f29ac5e68d4d8da8f3496fefe695ec P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.798830 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9]
I20250905 19:52:39.799494 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6e9009b9124c413eb5058a1f3294990a [id=78d19980c7cc4a79b1b1dccc0a7f674a]
I20250905 19:52:39.800046 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b]
I20250905 19:52:39.801213 3151 consensus_queue.cc:1035] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:39.806571 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu pid 1262: Timed out: CheckLeaks RPC to 127.0.104.60:34395 timed out after 1.000s (SENT)
I20250905 19:52:39.806758 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu with pid 1262
I20250905 19:52:39.808681 3299 raft_consensus.cc:513] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.809144 906 raft_consensus.cc:2466] T 74eb2632a6414e52b9ad67f53c182679 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.809330 3299 leader_election.cc:290] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.809484 3299 consensus_queue.cc:1035] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.810045 986 leader_election.cc:304] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.810295 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e8bfe047a6e4cc087eea75244da5b41" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.810470 906 raft_consensus.cc:3058] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.810642 3299 raft_consensus.cc:2802] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.810756 3299 raft_consensus.cc:695] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.810982 3299 consensus_queue.cc:237] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.816352 3346 raft_consensus.cc:491] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4eb3af79b01d4835bf6bf7b26bf016f5)
I20250905 19:52:39.816491 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7e8bfe047a6e4cc087eea75244da5b41" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:39.816504 3346 raft_consensus.cc:513] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.817138 3346 leader_election.cc:290] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:39.817706 984 leader_election.cc:343] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:39.818176 712 leader_election.cc:343] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3c824f5591044c4198a51a25661f71e5
W20250905 19:52:39.818600 711 leader_election.cc:343] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3c824f5591044c4198a51a25661f71e5
I20250905 19:52:39.818701 711 leader_election.cc:304] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:39.819043 3346 raft_consensus.cc:2747] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:39.827557 3151 consensus_queue.cc:1035] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.831243 766 raft_consensus.cc:3058] T 5aff2acc3f8245e1bca41f971f8e74ce P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.834419 1169 tablet_bootstrap.cc:492] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.834578 1169 ts_tablet_manager.cc:1397] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.076s user 0.017s sys 0.008s
I20250905 19:52:39.835294 1169 raft_consensus.cc:357] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.835479 1169 raft_consensus.cc:383] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.835544 1169 raft_consensus.cc:738] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.835717 1170 tablet_bootstrap.cc:492] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.835763 1169 consensus_queue.cc:260] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.836006 1170 ts_tablet_manager.cc:1397] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.192s user 0.003s sys 0.005s
I20250905 19:52:39.836297 1169 ts_tablet_manager.cc:1428] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:39.836565 1169 tablet_bootstrap.cc:492] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.836910 1170 raft_consensus.cc:357] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.837145 1170 raft_consensus.cc:383] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.837263 1170 raft_consensus.cc:738] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.838668 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef]
I20250905 19:52:39.842583 1169 tablet_bootstrap.cc:654] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.846586 1170 consensus_queue.cc:260] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.848058 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4]
I20250905 19:52:39.850330 1170 ts_tablet_manager.cc:1428] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.014s user 0.001s sys 0.001s
I20250905 19:52:39.851115 1170 tablet_bootstrap.cc:492] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.853590 1170 tablet_bootstrap.cc:654] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.858345 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9]
I20250905 19:52:39.869669 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31]
I20250905 19:52:39.870807 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6]
I20250905 19:52:39.881062 1169 tablet_bootstrap.cc:492] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.881223 1169 ts_tablet_manager.cc:1397] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.045s user 0.007s sys 0.006s
I20250905 19:52:39.881924 1169 raft_consensus.cc:357] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.882103 1169 raft_consensus.cc:383] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.882153 1169 raft_consensus.cc:738] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.882336 1169 consensus_queue.cc:260] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.882843 1169 ts_tablet_manager.cc:1428] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:39.883105 1169 tablet_bootstrap.cc:492] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:39.884579 1169 tablet_bootstrap.cc:654] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.898758 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716]
I20250905 19:52:39.899858 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_00d9a0127df4444eb081316b26c6ec87 [id=820565dc3d0942839324e01101b02e11]
I20250905 19:52:39.908263 906 raft_consensus.cc:2466] T 7e8bfe047a6e4cc087eea75244da5b41 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:39.908782 3282 raft_consensus.cc:513] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.909027 3274 raft_consensus.cc:491] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.909161 3274 raft_consensus.cc:513] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.909564 3282 leader_election.cc:290] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.909694 3274 leader_election.cc:290] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.912040 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "414f30a772524c33a74864ff741c865f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.912665 984 leader_election.cc:343] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.915899 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "414f30a772524c33a74864ff741c865f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.916182 906 raft_consensus.cc:2466] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.916689 986 leader_election.cc:304] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.918306 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbd471695e748c9831f9b0f6795aea7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:39.918545 1048 raft_consensus.cc:3058] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.919262 3274 raft_consensus.cc:2802] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:39.919420 3274 raft_consensus.cc:695] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:39.919749 3274 consensus_queue.cc:237] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.922045 986 leader_election.cc:304] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.922497 3297 raft_consensus.cc:2802] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.922513 1170 tablet_bootstrap.cc:492] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.922574 3297 raft_consensus.cc:491] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.922647 1170 ts_tablet_manager.cc:1397] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.072s user 0.002s sys 0.002s
I20250905 19:52:39.922641 3297 raft_consensus.cc:3058] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.923290 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5cbd471695e748c9831f9b0f6795aea7" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.923367 1170 raft_consensus.cc:357] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.923570 1170 raft_consensus.cc:383] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.923637 1170 raft_consensus.cc:738] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.923821 907 raft_consensus.cc:3058] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.923839 1170 consensus_queue.cc:260] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.924584 1170 ts_tablet_manager.cc:1428] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:39.925678 3151 raft_consensus.cc:491] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.925791 3151 raft_consensus.cc:513] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.926319 3151 leader_election.cc:290] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.929884 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:39.930469 984 leader_election.cc:343] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.934084 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.934268 906 raft_consensus.cc:2466] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.934999 986 leader_election.cc:304] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:39.935580 3151 raft_consensus.cc:2802] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.935673 3151 raft_consensus.cc:491] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.935751 3151 raft_consensus.cc:3058] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.941256 2496 raft_consensus.cc:513] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.942422 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e63b27ff4e434d76a8280f90c22f4082" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:39.942596 764 raft_consensus.cc:3058] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.942760 1170 tablet_bootstrap.cc:492] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.947096 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e63b27ff4e434d76a8280f90c22f4082" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:39.947350 906 raft_consensus.cc:3058] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.950047 1170 tablet_bootstrap.cc:654] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.959736 1170 tablet_bootstrap.cc:492] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.959913 1170 ts_tablet_manager.cc:1397] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.001s
I20250905 19:52:39.960062 3299 consensus_queue.cc:1035] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:39.960736 1170 raft_consensus.cc:357] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.960928 1170 raft_consensus.cc:383] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.960988 1170 raft_consensus.cc:738] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.961258 2496 leader_election.cc:290] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.961184 1170 consensus_queue.cc:260] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.961808 1170 ts_tablet_manager.cc:1428] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:39.962066 1170 tablet_bootstrap.cc:492] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.963914 1170 tablet_bootstrap.cc:654] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.973784 3274 raft_consensus.cc:491] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.973985 3274 raft_consensus.cc:513] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.974596 3274 leader_election.cc:290] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:39.975342 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.975557 909 raft_consensus.cc:2466] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:39.976306 986 leader_election.cc:304] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:39.976863 3274 raft_consensus.cc:2802] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.976867 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:39.976948 3274 raft_consensus.cc:491] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.977008 3274 raft_consensus.cc:3058] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:39.977360 984 leader_election.cc:343] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:39.980931 1170 tablet_bootstrap.cc:492] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:39.981086 1170 ts_tablet_manager.cc:1397] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.019s user 0.005s sys 0.002s
I20250905 19:52:39.981963 1170 raft_consensus.cc:357] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.982157 1170 raft_consensus.cc:383] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.982223 1170 raft_consensus.cc:738] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.982468 1170 consensus_queue.cc:260] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.983068 1170 ts_tablet_manager.cc:1428] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:39.983409 1170 tablet_bootstrap.cc:492] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:39.985352 1170 tablet_bootstrap.cc:654] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:39.980176 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]
I20250905 19:52:39.987787 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_5277901ee519486abb3c0e90d83961b5 [id=848f6c88009643beb5a2abcf7d5c35b7]
I20250905 19:52:39.988392 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e]
I20250905 19:52:39.991609 1169 tablet_bootstrap.cc:492] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:39.991783 1169 ts_tablet_manager.cc:1397] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.109s user 0.017s sys 0.021s
I20250905 19:52:39.992560 1169 raft_consensus.cc:357] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.992749 1169 raft_consensus.cc:383] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:39.992812 1169 raft_consensus.cc:738] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:39.993012 1169 consensus_queue.cc:260] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.995667 3282 raft_consensus.cc:491] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:39.995820 3282 raft_consensus.cc:513] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:39.996600 3282 leader_election.cc:290] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:39.996592 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]
I20250905 19:52:39.997251 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c870e689af254fb0acb15330577e5c42" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:39.997398 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c870e689af254fb0acb15330577e5c42" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:39.997460 909 raft_consensus.cc:2466] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:39.997565 1049 raft_consensus.cc:2466] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:39.997818 712 leader_election.cc:304] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:39.998256 3282 raft_consensus.cc:2802] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:39.998351 3282 raft_consensus.cc:491] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:39.998435 3282 raft_consensus.cc:3058] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:39.999357 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_487dad5063df425c8b4a44d7f8adbd39 [id=866aff163c794895abdd7632b46a252b]
I20250905 19:52:39.999873 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482]
I20250905 19:52:40.004359 3375 raft_consensus.cc:2747] T 5b4f9e12781a46f19b563746a70b59e4 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.004727 1048 raft_consensus.cc:2466] T 5cbd471695e748c9831f9b0f6795aea7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:40.005177 711 leader_election.cc:304] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:40.007365 3297 raft_consensus.cc:513] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.008277 2496 consensus_queue.cc:1035] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.009320 1170 tablet_bootstrap.cc:492] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.009446 1170 ts_tablet_manager.cc:1397] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.000s
I20250905 19:52:40.010282 1170 raft_consensus.cc:357] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.010439 1170 raft_consensus.cc:383] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.010483 1170 raft_consensus.cc:738] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.010649 1170 consensus_queue.cc:260] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.010811 3346 raft_consensus.cc:2802] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.016023 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "414f30a772524c33a74864ff741c865f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.016669 984 leader_election.cc:343] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.016770 767 raft_consensus.cc:3058] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.017913 2931 consensus_queue.cc:1035] T 5b4f9e12781a46f19b563746a70b59e4 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.018085 906 raft_consensus.cc:2466] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.018555 986 leader_election.cc:304] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.018817 3299 raft_consensus.cc:2802] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.018963 3299 raft_consensus.cc:695] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.019248 3299 consensus_queue.cc:237] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.023802 3297 leader_election.cc:290] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.024572 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "414f30a772524c33a74864ff741c865f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.024765 906 raft_consensus.cc:3058] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.032186 2422 raft_consensus.cc:491] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.032352 2422 raft_consensus.cc:513] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.032907 2422 leader_election.cc:290] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.033566 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.033881 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.034370 849 leader_election.cc:343] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:40.034734 848 leader_election.cc:343] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.034829 848 leader_election.cc:304] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.035176 2422 raft_consensus.cc:2747] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.040369 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde]
I20250905 19:52:40.046315 2422 raft_consensus.cc:491] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.046505 2422 raft_consensus.cc:513] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.047194 1169 ts_tablet_manager.cc:1428] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.055s user 0.002s sys 0.000s
I20250905 19:52:40.047199 2422 leader_election.cc:290] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.047520 1169 tablet_bootstrap.cc:492] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.048040 3346 raft_consensus.cc:695] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:40.048143 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.048345 3346 consensus_queue.cc:237] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:40.048931 849 leader_election.cc:343] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.049736 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.050277 848 leader_election.cc:343] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.050395 848 leader_election.cc:304] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.050794 2422 raft_consensus.cc:2747] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.055359 1169 tablet_bootstrap.cc:654] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.061426 3299 raft_consensus.cc:491] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.061599 3299 raft_consensus.cc:513] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.061674 2496 consensus_queue.cc:1035] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.062605 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.062791 3299 leader_election.cc:290] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.063004 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.063182 909 raft_consensus.cc:2466] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.063609 986 leader_election.cc:304] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:40.063772 984 leader_election.cc:343] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.064010 3299 raft_consensus.cc:2802] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.064141 3299 raft_consensus.cc:491] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.064268 3299 raft_consensus.cc:3058] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.067226 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]
I20250905 19:52:40.070356 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc]
I20250905 19:52:40.071414 1170 ts_tablet_manager.cc:1428] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.062s user 0.000s sys 0.005s
I20250905 19:52:40.071636 1169 tablet_bootstrap.cc:492] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.071775 1169 ts_tablet_manager.cc:1397] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250905 19:52:40.071797 1170 tablet_bootstrap.cc:492] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.074086 1170 tablet_bootstrap.cc:654] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.079568 2496 consensus_queue.cc:1035] T b1f46261822442e0a961d2d63bdb35c8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.082222 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c]
I20250905 19:52:40.085413 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]
I20250905 19:52:40.079628 1169 raft_consensus.cc:357] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.089076 1169 raft_consensus.cc:383] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.089175 1169 raft_consensus.cc:738] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.089417 1169 consensus_queue.cc:260] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.089473 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]
I20250905 19:52:40.089922 1169 ts_tablet_manager.cc:1428] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:40.090090 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]
I20250905 19:52:40.090191 1169 tablet_bootstrap.cc:492] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.090703 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6]
I20250905 19:52:40.092331 764 raft_consensus.cc:2466] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.092331 3151 raft_consensus.cc:513] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.092921 3151 leader_election.cc:290] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.093393 906 raft_consensus.cc:2466] T 414f30a772524c33a74864ff741c865f P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.094551 1169 tablet_bootstrap.cc:654] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.095038 3282 raft_consensus.cc:513] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.099382 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c870e689af254fb0acb15330577e5c42" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.099581 906 raft_consensus.cc:3058] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.100330 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.100343 986 leader_election.cc:304] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.100478 904 raft_consensus.cc:3058] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.100721 3151 raft_consensus.cc:2802] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.100754 3282 leader_election.cc:290] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.100862 3151 raft_consensus.cc:695] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.101087 3151 consensus_queue.cc:237] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.102113 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.102638 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c]
I20250905 19:52:40.102805 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c870e689af254fb0acb15330577e5c42" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:40.102977 1048 raft_consensus.cc:3058] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.103374 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]
W20250905 19:52:40.103726 984 leader_election.cc:343] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.105716 907 raft_consensus.cc:2466] T 5cbd471695e748c9831f9b0f6795aea7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:40.108654 906 raft_consensus.cc:2466] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:40.111493 904 raft_consensus.cc:2466] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.112385 2496 raft_consensus.cc:491] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.112514 2496 raft_consensus.cc:513] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.113065 2496 leader_election.cc:290] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.113677 712 leader_election.cc:304] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:40.114125 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.114683 986 leader_election.cc:304] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
W20250905 19:52:40.114742 984 leader_election.cc:343] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.115036 3282 raft_consensus.cc:2802] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.115198 3282 raft_consensus.cc:695] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:40.115295 2496 raft_consensus.cc:2802] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.115334 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07]
I20250905 19:52:40.115423 2496 raft_consensus.cc:695] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.115639 2496 consensus_queue.cc:237] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.115926 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09]
I20250905 19:52:40.116462 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]
I20250905 19:52:40.116891 1170 tablet_bootstrap.cc:492] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.117039 1170 ts_tablet_manager.cc:1397] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.001s
I20250905 19:52:40.117161 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b]
I20250905 19:52:40.117174 3274 raft_consensus.cc:513] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.117697 3274 leader_election.cc:290] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.118017 3299 raft_consensus.cc:513] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.118467 3299 leader_election.cc:290] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.120959 1048 raft_consensus.cc:2466] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:40.121084 3430 raft_consensus.cc:491] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.121206 3430 raft_consensus.cc:513] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.123679 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.123863 907 raft_consensus.cc:3058] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.125321 1169 tablet_bootstrap.cc:492] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.125458 1169 ts_tablet_manager.cc:1397] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.035s user 0.011s sys 0.000s
I20250905 19:52:40.126345 1169 raft_consensus.cc:357] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.126866 1169 raft_consensus.cc:383] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.126879 907 raft_consensus.cc:2466] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.126919 1169 raft_consensus.cc:738] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.127138 1169 consensus_queue.cc:260] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.127578 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18]
I20250905 19:52:40.127630 1169 ts_tablet_manager.cc:1428] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:40.127945 1169 tablet_bootstrap.cc:492] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.128109 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]
I20250905 19:52:40.128307 986 leader_election.cc:304] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.128597 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9]
I20250905 19:52:40.128566 3297 consensus_queue.cc:1035] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.129025 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_fe3dee104cc640359210483bc31532cf [id=99384aac4e624d6db630b249c3ba2543]
I20250905 19:52:40.129103 3297 raft_consensus.cc:2802] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.129195 3297 raft_consensus.cc:695] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.129464 3297 consensus_queue.cc:237] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.129729 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40]
I20250905 19:52:40.129901 1169 tablet_bootstrap.cc:654] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.130172 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]
I20250905 19:52:40.130597 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]
I20250905 19:52:40.135763 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.136044 904 raft_consensus.cc:2466] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.136499 986 leader_election.cc:304] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.136988 3151 raft_consensus.cc:2802] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.137076 3151 raft_consensus.cc:491] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.137152 3151 raft_consensus.cc:3058] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.141911 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.142143 906 raft_consensus.cc:3058] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.149729 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778]
I20250905 19:52:40.150401 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]
I20250905 19:52:40.150406 906 raft_consensus.cc:2466] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.150852 986 leader_election.cc:304] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.150962 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959]
I20250905 19:52:40.151181 3274 raft_consensus.cc:2802] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.151326 3274 raft_consensus.cc:695] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.151679 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]
I20250905 19:52:40.151587 3274 consensus_queue.cc:237] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.152542 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0]
I20250905 19:52:40.155381 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.156215 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.156673 984 leader_election.cc:343] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:40.157021 984 leader_election.cc:343] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.157656 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7]
I20250905 19:52:40.164619 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30]
I20250905 19:52:40.164826 3430 leader_election.cc:290] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.155753 3282 consensus_queue.cc:237] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.167671 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3]
I20250905 19:52:40.167723 1170 raft_consensus.cc:357] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.167938 1170 raft_consensus.cc:383] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.168023 1170 raft_consensus.cc:738] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.168228 1170 consensus_queue.cc:260] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.168797 1170 ts_tablet_manager.cc:1428] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.052s user 0.003s sys 0.000s
I20250905 19:52:40.169391 1170 tablet_bootstrap.cc:492] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.169941 3299 consensus_queue.cc:1035] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.171342 1170 tablet_bootstrap.cc:654] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.171979 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d]
I20250905 19:52:40.172757 1169 tablet_bootstrap.cc:492] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.173302 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d]
W20250905 19:52:40.177213 711 leader_election.cc:343] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:40.177498 1169 ts_tablet_manager.cc:1397] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.050s user 0.002s sys 0.011s
I20250905 19:52:40.178325 1169 raft_consensus.cc:357] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.179093 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_56ae03b8d3eb4a58970a492506fcc6f1 [id=acf1b6990a594051ba94b43420cdeebc]
I20250905 19:52:40.184566 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e]
I20250905 19:52:40.180253 2496 consensus_queue.cc:1035] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.185139 1169 raft_consensus.cc:383] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.185269 1169 raft_consensus.cc:738] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
W20250905 19:52:40.185636 1110 connection.cc:537] client connection to 127.0.104.60:34395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:40.186646 1110 master_proxy_rpc.cc:203] Re-attempting ReplaceTablet request to leader Master (127.0.104.60:34395)
W20250905 19:52:40.187063 1110 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.60:34395)
W20250905 19:52:40.187875 1110 master_proxy_rpc.cc:203] Re-attempting DeleteTable request to leader Master (127.0.104.60:34395)
W20250905 19:52:40.188843 1110 master_proxy_rpc.cc:203] Re-attempting AlterTable request to leader Master (127.0.104.60:34395)
W20250905 19:52:40.190459 712 leader_election.cc:343] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:40.190630 712 leader_election.cc:304] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:40.191035 3282 raft_consensus.cc:2747] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:40.201992 712 connection.cc:537] server connection from 127.0.0.1:54786 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:40.202026 512 connection.cc:537] client connection to 127.0.104.60:34395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:40.202260 512 leader_election.cc:336] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:40.202416 512 leader_election.cc:304] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: c6c0f2b9d8cd457a9c3448034c3d5c91; no voters: c401c9f3020b4ae5931c566c6bcb1711, e38333af29db4bf6af41ee5a25d6701e
I20250905 19:52:40.202834 3412 raft_consensus.cc:2747] T 00000000000000000000000000000000 P c6c0f2b9d8cd457a9c3448034c3d5c91 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20250905 19:52:40.203711 985 connection.cc:537] client connection to 127.0.104.60:34395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:40.188107 2291 connection.cc:537] client connection to 127.0.104.60:34395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20250905 19:52:40.204298 2291 leader_election.cc:336] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 4 election: RPC error from VoteRequest() call to peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:40.204619 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_96c633b175ac41309951506efe002443 [id=ae92837e2ac94949aac8822f084c0a97]
W20250905 19:52:40.204751 2291 leader_election.cc:336] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 [CANDIDATE]: Term 4 pre-election: RPC error from VoteRequest() call to peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:40.205303 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]
I20250905 19:52:40.206151 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d]
I20250905 19:52:40.206858 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d]
I20250905 19:52:40.202524 1169 consensus_queue.cc:260] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:40.207249 849 connection.cc:537] client connection to 127.0.104.60:34395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:40.207844 1169 ts_tablet_manager.cc:1428] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.030s user 0.001s sys 0.001s
W20250905 19:52:40.209229 711 connection.cc:537] client connection to 127.0.104.60:34395 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20250905 19:52:40.209980 1169 tablet_bootstrap.cc:492] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.212718 1169 tablet_bootstrap.cc:654] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.205673 2072 master_runner.cc:186] Error getting master registration for 127.0.104.60:34395: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107),
I20250905 19:52:40.218734 2072 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20250905 19:52:40.218886 2291 consensus_peers.cc:489] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 -> Peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Couldn't send request to peer e38333af29db4bf6af41ee5a25d6701e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:40.219764 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6]
I20250905 19:52:40.232331 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_af469c2f14f24b01ae62bc25f1e8e4f8 [id=ba5be2a2e05e47e8a32ae828f6c828e6]
I20250905 19:52:40.233716 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0 [id=beb46dbcc0634881b6c69db37796ce3c]
I20250905 19:52:40.234308 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]
I20250905 19:52:40.234839 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_19682dbce0bc4480b82d39235ba7f55b [id=bff29fe995d34f32ad9780ad5591a8b2]
I20250905 19:52:40.235412 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e02928704120437bbdda2c9819761a85 [id=c08a19b2c2384f52b2f91b47ef74e13f]
I20250905 19:52:40.238348 1169 tablet_bootstrap.cc:492] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.238617 1169 ts_tablet_manager.cc:1397] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20250905 19:52:40.239665 1169 raft_consensus.cc:357] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.240051 1169 raft_consensus.cc:383] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.240170 1169 raft_consensus.cc:738] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.240480 1169 consensus_queue.cc:260] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.241443 1169 ts_tablet_manager.cc:1428] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:40.241791 1169 tablet_bootstrap.cc:492] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.243323 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5]
I20250905 19:52:40.243947 1169 tablet_bootstrap.cc:654] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.244011 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]
I20250905 19:52:40.244596 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_3c5a0a7b75424d2ab2975a941bab388b [id=c3c7d5d0e4394865a0488381d8cb4c97]
I20250905 19:52:40.245102 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]
I20250905 19:52:40.140362 3151 raft_consensus.cc:513] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.246744 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84]
I20250905 19:52:40.247006 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.247748 984 leader_election.cc:343] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.248206 3151 leader_election.cc:290] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.249920 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.250264 904 raft_consensus.cc:3058] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.252748 1169 tablet_bootstrap.cc:492] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.252887 1169 ts_tablet_manager.cc:1397] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.011s user 0.001s sys 0.003s
I20250905 19:52:40.253700 904 raft_consensus.cc:2466] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.254129 3346 raft_consensus.cc:491] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.254410 986 leader_election.cc:304] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.254370 3346 raft_consensus.cc:513] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.255187 3151 raft_consensus.cc:2802] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.255347 3346 leader_election.cc:290] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.255411 3151 raft_consensus.cc:695] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.255661 3151 consensus_queue.cc:237] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.256889 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_37507bfcf78648779f20bfe755073f99 [id=cbe19099fb774bdf8204799fb6ce4243]
I20250905 19:52:40.257831 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f]
I20250905 19:52:40.258467 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73]
I20250905 19:52:40.259086 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2]
I20250905 19:52:40.260367 764 raft_consensus.cc:3058] T a860ab3f5ce54bd2b75bb71730a3c6e6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.253655 1169 raft_consensus.cc:357] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.264952 1169 raft_consensus.cc:383] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.265051 1169 raft_consensus.cc:738] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.265288 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb]
I20250905 19:52:40.266532 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_57684ee2f11b44129e0b3d35029151fc [id=d636c310314a405e8d4f99e56d94f44c]
I20250905 19:52:40.266265 1169 consensus_queue.cc:260] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.267305 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]
I20250905 19:52:40.268426 1169 ts_tablet_manager.cc:1428] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250905 19:52:40.268746 1169 tablet_bootstrap.cc:492] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.268886 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]
I20250905 19:52:40.270859 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348]
I20250905 19:52:40.272086 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1]
I20250905 19:52:40.275269 1169 tablet_bootstrap.cc:654] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.278244 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_5bf03647f71d46a8b3d3f0b5bc385154 [id=e19dba4ecf6d4c22965720b89e7c45d4]
I20250905 19:52:40.279907 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530]
I20250905 19:52:40.284929 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1]
I20250905 19:52:40.288220 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151]
I20250905 19:52:40.288929 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a]
I20250905 19:52:40.292645 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244]
I20250905 19:52:40.293285 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_6a2d2733af51406295e2d14f82f6cb1e [id=edb5d0cff27f4181a256a126d5319d55]
I20250905 19:52:40.303794 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816]
I20250905 19:52:40.304852 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d]
I20250905 19:52:40.309864 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39]
I20250905 19:52:40.310963 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4]
I20250905 19:52:40.311586 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801]
I20250905 19:52:40.312322 1169 tablet_bootstrap.cc:492] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.312463 1169 ts_tablet_manager.cc:1397] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.000s
I20250905 19:52:40.313205 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e]
I20250905 19:52:40.313184 1169 raft_consensus.cc:357] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.313356 1169 raft_consensus.cc:383] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.313410 1169 raft_consensus.cc:738] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.321004 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]
I20250905 19:52:40.321478 1169 consensus_queue.cc:260] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.321836 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]
I20250905 19:52:40.323139 1169 ts_tablet_manager.cc:1428] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:40.323436 1169 tablet_bootstrap.cc:492] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.323657 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]
I20250905 19:52:40.324301 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147]
I20250905 19:52:40.328394 1169 tablet_bootstrap.cc:654] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.329074 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac]
I20250905 19:52:40.329787 3366 catalog_manager.cc:671] Loaded metadata for table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7]
I20250905 19:52:40.339531 1170 tablet_bootstrap.cc:492] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.339702 1170 ts_tablet_manager.cc:1397] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.170s user 0.035s sys 0.095s
I20250905 19:52:40.344656 1170 raft_consensus.cc:357] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.344903 1170 raft_consensus.cc:383] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.345002 1170 raft_consensus.cc:738] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.345227 1170 consensus_queue.cc:260] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.345361 3366 tablet_loader.cc:96] loaded metadata for tablet 0039b8f0ba704907b2820c9e3a69cd73 (table default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a])
I20250905 19:52:40.345764 1170 ts_tablet_manager.cc:1428] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250905 19:52:40.346113 1170 tablet_bootstrap.cc:492] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.346355 3366 tablet_loader.cc:96] loaded metadata for tablet 00d1d24868204ce09a7f5e40357fec18 (table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62])
I20250905 19:52:40.347995 3366 tablet_loader.cc:96] loaded metadata for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46])
I20250905 19:52:40.348452 3366 tablet_loader.cc:96] loaded metadata for tablet 040e65fdb1104b07b953044f9b6b538a (table default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a])
I20250905 19:52:40.349289 3366 tablet_loader.cc:96] loaded metadata for tablet 05ec559d0f83445892f8285f9bdeb2c8 (table default.Table_45e1bab15bd84e51a1a749ebfd98eeab [id=f5891226b675435f81cbdca3c8b4f43e])
I20250905 19:52:40.349943 3366 tablet_loader.cc:96] loaded metadata for tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03])
I20250905 19:52:40.351258 3366 tablet_loader.cc:96] loaded metadata for tablet 0d93a328d4934e5294ec66f739eb240b (table default.Table_10296fdc41264234b254f3e5ce2be23d [id=d0571b6e556f4be3870d0875997d7ea2])
I20250905 19:52:40.351670 3366 tablet_loader.cc:96] loaded metadata for tablet 0e05db608a154b1ca0f9a1ed06c7fbd3 (table default.Table_5f2927da18ef4c51a2b70428fb215a76 [id=fdf4d991177c459b9a25308aac282147])
I20250905 19:52:40.352298 3366 tablet_loader.cc:96] loaded metadata for tablet 0fb8fe93061848a0b055979bad10f08b (table default.Table_708e16c265ea413f86dbd083d7b9dcde [id=4925822c977c4a4588c9fe6117ee722e])
I20250905 19:52:40.352723 3366 tablet_loader.cc:96] loaded metadata for tablet 10fef848b9b746c5a0de6a8c3ca74f7a (table default.Table_94f58de97c5946149d23d635eda4d555 [id=656d24e0a3ad4ac78696df5aec2f279e])
I20250905 19:52:40.353116 3366 tablet_loader.cc:96] loaded metadata for tablet 1110897f9ec14741b9b882b6e09081ab (table default.Table_b24d9f16949145cda6b4f82d78993fa0 [id=2242efd05b0e4fae989ce99e9431aa4b])
I20250905 19:52:40.353771 3366 tablet_loader.cc:96] loaded metadata for tablet 160a7814abf9423dbe965c252c20fa7f (table default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00])
I20250905 19:52:40.354653 3366 tablet_loader.cc:96] loaded metadata for tablet 180437d1534946619384307ddcb7c937 (table default.Table_c14282695b114ab4b1d2e98d4d6d78fb [id=8b539ba054ec4de4bcdd0b8e3f5455fc])
I20250905 19:52:40.355401 3366 tablet_loader.cc:96] loaded metadata for tablet 1bd731a303a545f492b19f774520582c (table default.Table_6016725fc2e44e0381a7cc4e06bcac86 [id=ed7ed5bb46a84eb2bc77d1f948592244])
I20250905 19:52:40.356055 3366 tablet_loader.cc:96] loaded metadata for tablet 1d095804b51b4205a23ac3d3106312f2 (table default.Table_869aaac5cd2744e3b23de6c1060391a9 [id=8996c7d409a24ffb8de95268093f3bde])
I20250905 19:52:40.356705 3366 tablet_loader.cc:96] loaded metadata for tablet 1e9b7458f96148b08962a9b82223187e (table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311])
I20250905 19:52:40.357707 3366 tablet_loader.cc:96] loaded metadata for tablet 239539ac19124de192cadae494720a2b (table default.Table_c2a680ef01e34dceb09eb62c5948d981 [id=cf56ce3a388047dc9bd2d846817fef73])
I20250905 19:52:40.358146 3366 tablet_loader.cc:96] loaded metadata for tablet 2466a1bafc8347e483ca5e28d552bfc9 (table default.Table_c30c09bc294d465da486648091481ed1 [id=3118b283e4194c5fa5c5a7e270d14e49])
W20250905 19:52:40.358500 3453 negotiation.cc:337] Failed RPC negotiation. Trace:
0905 19:52:40.190268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:34395 (local address 127.0.0.1:57236)
0905 19:52:40.357806 (+167538us) negotiation.cc:107] Waiting for socket to connect
0905 19:52:40.357950 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection reset by peer (error 104)
Metrics: {"client-negotiator.queue_time_us":167353,"thread_start_us":167246,"threads_started":1}
I20250905 19:52:40.358760 3366 tablet_loader.cc:96] loaded metadata for tablet 2568b9f6f37b48efbe92cef4c19d5dfb (table default.Table_8a59f0ac61434a5880164439529876f4 [id=94dad5d2c24149afb560da14e1ec7c7b])
I20250905 19:52:40.359416 3366 tablet_loader.cc:96] loaded metadata for tablet 2642bb409bb4473db034035b1b834739 (table default.Table_cfe02f65c42d492b9ebd6d9891ff4b34 [id=03ed72cf63184003be5e02061d130ce3])
I20250905 19:52:40.359542 1169 tablet_bootstrap.cc:492] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.359695 1169 ts_tablet_manager.cc:1397] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.036s user 0.006s sys 0.000s
I20250905 19:52:40.360387 3366 tablet_loader.cc:96] loaded metadata for tablet 298a325f5e4645beac39eb59ddc0566d (table default.Table_44734742f0c948af9cce54466ff759c6 [id=11511883d6b046c7b155ff9687eb2c76])
I20250905 19:52:40.360546 1169 raft_consensus.cc:357] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.360735 1169 raft_consensus.cc:383] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.360803 1169 raft_consensus.cc:738] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.361030 1169 consensus_queue.cc:260] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.361518 1169 ts_tablet_manager.cc:1428] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:40.361857 1169 tablet_bootstrap.cc:492] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.362684 3366 tablet_loader.cc:96] loaded metadata for tablet 30e1ca2bd1f94f748f2022b5669bc07e (table default.Table_85e1b8aa7bd340459631f0b02ae94b9d [id=151cf052220f4749be3757d20864c425])
I20250905 19:52:40.363191 1170 tablet_bootstrap.cc:654] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.363314 3366 tablet_loader.cc:96] loaded metadata for tablet 31484264112d42b089536d1ecc8ba021 (table default.Table_37507bfcf78648779f20bfe755073f99 [id=cbe19099fb774bdf8204799fb6ce4243])
I20250905 19:52:40.363730 3366 tablet_loader.cc:96] loaded metadata for tablet 31e332cdfedd4ff6b5026baa73e2caf3 (table default.Table_247bc51043654b48925883ad1c002451 [id=a65595f577e14f49b0c96e23db760c30])
I20250905 19:52:40.364492 3366 tablet_loader.cc:96] loaded metadata for tablet 32fa16972f2a42d39b338b833c185b28 (table default.Table_07c414d9fb004a4086d372ed79538be1 [id=a39e22fd5a194977be8bac4c5e4f0959])
I20250905 19:52:40.364943 3366 tablet_loader.cc:96] loaded metadata for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_d1234b81a24b4cdfbcfe61677e6bb957 [id=9c5a9ace09234365a465b92fd6bb7f40])
I20250905 19:52:40.365360 3366 tablet_loader.cc:96] loaded metadata for tablet 3587780819704008b7f4ee6abc714db6 (table default.Table_7d2ebfe52fb74920893aea327256e248 [id=62c0073253ab48cf863e26d4a09aa3f1])
I20250905 19:52:40.365679 1169 tablet_bootstrap.cc:654] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.366469 3366 tablet_loader.cc:96] loaded metadata for tablet 37311372c03644b0a99ce92b5d42bfd1 (table default.Table_10627325344c469294d0ea3e9036daea [id=4f107feeaf0c440c9d7b9d9cfd8be81c])
I20250905 19:52:40.366859 3366 tablet_loader.cc:96] loaded metadata for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 (table default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96])
I20250905 19:52:40.367923 3366 tablet_loader.cc:96] loaded metadata for tablet 3b3bcd5a4f9c46e99664f6cbe9167301 (table default.Table_3c5a0a7b75424d2ab2975a941bab388b [id=c3c7d5d0e4394865a0488381d8cb4c97])
I20250905 19:52:40.368356 3366 tablet_loader.cc:96] loaded metadata for tablet 3b53259c71d5443ebb12827ee78038fe (table default.Table_45e2499e36e8452fbe6d4d39b5adbe1d [id=a8e6436fd67548b68b12c258c47ffff3])
I20250905 19:52:40.368706 3366 tablet_loader.cc:96] loaded metadata for tablet 3c5cf2bc19b043f68838f8a6af6393cd (table default.Table_c6be55785e1244cb85764306f76c04a2 [id=5ee8596f32f042698dfbe530710fda20])
I20250905 19:52:40.369325 3366 tablet_loader.cc:96] loaded metadata for tablet 3d3a7b03d5e34f8dbcd5b6b994c9b419 (table default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3 [id=3b2cd4d8f9964d0588a638cd28af56b1])
I20250905 19:52:40.369993 3366 tablet_loader.cc:96] loaded metadata for tablet 3f52ff57fb1a4ea8a1ad4d5f0480c7f2 (table default.Table_7f5bdee0d6b34929b57b46c6055c4ad6 [id=ad1850fef5004524ad4e555f51d21d8e])
I20250905 19:52:40.370425 3366 tablet_loader.cc:96] loaded metadata for tablet 3fe2b75a3e59437c885abc18614f1da2 (table default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348])
I20250905 19:52:40.370838 3366 tablet_loader.cc:96] loaded metadata for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7])
I20250905 19:52:40.371218 3366 tablet_loader.cc:96] loaded metadata for tablet 415508e72ee34bc391c4369b5f8a2f48 (table default.Table_7afe807a480844efac1cb503fbba94ae [id=92aa73bcd0ad43d69787ac8ed44d6a07])
I20250905 19:52:40.371634 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.373364 3366 tablet_loader.cc:96] loaded metadata for tablet 48be704b238e481089c727b3e1b96a2f (table default.Table_e20d0ff772494c13b5e9aa353a1da157 [id=7d51c5d8d58641dda8e8c44a1bcaedf6])
I20250905 19:52:40.373983 3366 tablet_loader.cc:96] loaded metadata for tablet 4a518797d26140f5b9abe41eab415040 (table default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3])
I20250905 19:52:40.374519 3366 tablet_loader.cc:96] loaded metadata for tablet 4a95130c82e14bbcae577d4244b876ab (table default.Table_00d9a0127df4444eb081316b26c6ec87 [id=820565dc3d0942839324e01101b02e11])
I20250905 19:52:40.374926 3366 tablet_loader.cc:96] loaded metadata for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (table default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776])
I20250905 19:52:40.376268 3366 tablet_loader.cc:96] loaded metadata for tablet 4d6f2e44855d4312b1c0367145e3cee8 (table default.Table_024f98676ba34eaca458c9d9636e3b83 [id=7d085917cc0348c2812c8107e0ad6ca9])
I20250905 19:52:40.376711 3366 tablet_loader.cc:96] loaded metadata for tablet 4dd2a96b8180438fbc722be1c18a0189 (table default.Table_934044a3a6e849528a30d212f3dc5fd5 [id=8efb2e43c21b4fd6883a5d4176a995e6])
I20250905 19:52:40.376724 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "af10abf0a5b34b3bbfea20e5c2005006" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.377211 3366 tablet_loader.cc:96] loaded metadata for tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9])
I20250905 19:52:40.398825 2931 raft_consensus.cc:491] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.398970 2931 raft_consensus.cc:513] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.399189 3366 tablet_loader.cc:96] loaded metadata for tablet 4fac0e1f483b40ab8d66cb5a9ee808fc (table default.Table_9a54ba37e2a941ecaf67d8e56bc940f9 [id=7d3f5fcdae5f4e7dbebc80dc7a813b31])
I20250905 19:52:40.399513 2931 leader_election.cc:290] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.399787 3366 tablet_loader.cc:96] loaded metadata for tablet 51d92f92940b4952b68becf86661d459 (table default.Table_5277901ee519486abb3c0e90d83961b5 [id=848f6c88009643beb5a2abcf7d5c35b7])
I20250905 19:52:40.399849 2931 raft_consensus.cc:491] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.399945 2931 raft_consensus.cc:513] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.400414 3366 tablet_loader.cc:96] loaded metadata for tablet 5302b79208494e48a5f2778c5f52ad01 (table default.Table_b71a53bad9454675a843d031c9f776bf [id=dca199494b134720bfabd2162d077ae1])
I20250905 19:52:40.400450 2931 leader_election.cc:290] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.400792 3366 tablet_loader.cc:96] loaded metadata for tablet 539f67fac93c46ebb5897ea9f02016fd (table default.Table_af469c2f14f24b01ae62bc25f1e8e4f8 [id=ba5be2a2e05e47e8a32ae828f6c828e6])
I20250905 19:52:40.401409 3366 tablet_loader.cc:96] loaded metadata for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_1f75eecc8c8941a6aa9982f4e8cc6450 [id=4044052e369843d39980a31b1654628d])
I20250905 19:52:40.402019 3366 tablet_loader.cc:96] loaded metadata for tablet 55cecf59d3f24db88bb10b8e12c12f82 (table default.Table_0d6fe65405f543d7a70643f94bf22da5 [id=46715848277847218d557157e4dfc77b])
I20250905 19:52:40.403345 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.403951 984 leader_election.cc:343] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.404026 1170 tablet_bootstrap.cc:492] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.404168 1170 ts_tablet_manager.cc:1397] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.058s user 0.004s sys 0.000s
I20250905 19:52:40.404908 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edb9379335ca4eb5a32a6cbe85601a1f" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.405176 3366 tablet_loader.cc:96] loaded metadata for tablet 59c13a5b598a4597931ddf5d476aec85 (table default.Table_cb111dc4aa3946e8a89529d1cd623c08 [id=5ac15645cb4d4730a4b4464a85299bed])
W20250905 19:52:40.405385 984 leader_election.cc:343] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.406013 3366 tablet_loader.cc:96] loaded metadata for tablet 5a59d7fb5dc54b479d965282b1c7d24b (table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d])
I20250905 19:52:40.406258 1169 tablet_bootstrap.cc:492] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.406397 1169 ts_tablet_manager.cc:1397] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.045s user 0.002s sys 0.001s
I20250905 19:52:40.406591 3366 tablet_loader.cc:96] loaded metadata for tablet 5a5d38136c034b8497045176f749a98d (table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a])
I20250905 19:52:40.407270 3366 tablet_loader.cc:96] loaded metadata for tablet 5aff2acc3f8245e1bca41f971f8e74ce (table default.Table_f1ff5bb8b0734e03bca82540e14ee739 [id=53d907456fa84900a44d7598d656ad6a])
I20250905 19:52:40.407189 1169 raft_consensus.cc:357] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.407348 1169 raft_consensus.cc:383] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.407428 1169 raft_consensus.cc:738] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.407655 1169 consensus_queue.cc:260] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.408150 3366 tablet_loader.cc:96] loaded metadata for tablet 5b4f9e12781a46f19b563746a70b59e4 (table default.Table_cc71e9d45e1b4857a7f86f100e2d6376 [id=a186b82948d94293968444030cff4778])
I20250905 19:52:40.408216 1169 ts_tablet_manager.cc:1428] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:40.408865 3366 tablet_loader.cc:96] loaded metadata for tablet 5cbd471695e748c9831f9b0f6795aea7 (table default.Table_33f56154c6394adf9d27effde71d7363 [id=0100ae9ea57a4d6ba17167c6413a8a3a])
W20250905 19:52:40.408910 986 leader_election.cc:343] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:40.409044 986 leader_election.cc:304] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.409654 3366 tablet_loader.cc:96] loaded metadata for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50])
I20250905 19:52:40.409705 2931 raft_consensus.cc:2747] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.410099 3366 tablet_loader.cc:96] loaded metadata for tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b4a0d3409e7648df827261b499b9f127 [id=f0398b74058b4ccbac71a2cb2f584a8d])
I20250905 19:52:40.410733 3366 tablet_loader.cc:96] loaded metadata for tablet 61667a379974487195f81043683b9a24 (table default.Table_25d956aeed244b4d89f84d8f43086c55 [id=26cf482b4d50422eb27db543a7723f2f])
I20250905 19:52:40.411201 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.411458 904 raft_consensus.cc:2466] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.411612 3366 tablet_loader.cc:96] loaded metadata for tablet 65368306f86845008d4d486023e1fc29 (table default.Table_a73510a5e598492c83e3a2d2ee35d95d [id=33825b39a9f642349299783f18a68bf6])
I20250905 19:52:40.411882 986 leader_election.cc:304] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.412159 3366 tablet_loader.cc:96] loaded metadata for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e])
I20250905 19:52:40.412245 2931 raft_consensus.cc:2802] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.412319 2931 raft_consensus.cc:491] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.412370 2931 raft_consensus.cc:3058] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.413410 3366 tablet_loader.cc:96] loaded metadata for tablet 6b8bbbcac06944469e13f14ab1d80402 (table default.Table_36a3196459524d63aafe6ab49f9faf56 [id=a53bad170d7e4f12b3ed7588910cf7e0])
I20250905 19:52:40.414404 3366 tablet_loader.cc:96] loaded metadata for tablet 6e15350150db413fa4f100171280f46e (table default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff])
I20250905 19:52:40.414702 3366 tablet_loader.cc:96] loaded metadata for tablet 6e3e1732bff14d49ade14fbfa6a9784c (table default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d])
I20250905 19:52:40.415010 3366 tablet_loader.cc:96] loaded metadata for tablet 6e89d23c4cb64f568b25f92a278b9d5e (table default.Table_801aaff2487e46b3b2e0b28b3c0f83dd [id=853047bedbb242d893b5be8b5a5c156e])
I20250905 19:52:40.414985 2931 raft_consensus.cc:513] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.415566 3366 tablet_loader.cc:96] loaded metadata for tablet 6edbc312fc4947a79e937dbc64d30b37 (table default.Table_56ae03b8d3eb4a58970a492506fcc6f1 [id=acf1b6990a594051ba94b43420cdeebc])
I20250905 19:52:40.415907 3366 tablet_loader.cc:96] loaded metadata for tablet 6ee9f4fe3a464abaaca48f22ea183c0e (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39])
I20250905 19:52:40.416905 3366 tablet_loader.cc:96] loaded metadata for tablet 736bfce682e64498a8dde3310305536a (table default.Table_6a2d2733af51406295e2d14f82f6cb1e [id=edb5d0cff27f4181a256a126d5319d55])
I20250905 19:52:40.417583 3366 tablet_loader.cc:96] loaded metadata for tablet 74eb2632a6414e52b9ad67f53c182679 (table default.Table_4797854b3d9244bd8aac427a5de6985a [id=0919fd518eeb4b2fac8f9b910975a906])
I20250905 19:52:40.418087 2931 leader_election.cc:290] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.418623 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.418850 904 raft_consensus.cc:3058] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.419266 3366 tablet_loader.cc:96] loaded metadata for tablet 7a104b1c8ce14fa2bf101e1b34183cfa (table default.Table_6e40598f9dbc40ac9860eb5b78130a93 [id=408552212a1f42089f1d4cb87b4a13b5])
I20250905 19:52:40.419435 1169 tablet_bootstrap.cc:492] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.420466 3366 tablet_loader.cc:96] loaded metadata for tablet 7cdd4de055864d4fa6adb36bf190e609 (table default.Table_619041804f364e1d8be97fbf6e0ed649 [id=9810f826e4034ab7b6459b545ee4cf18])
I20250905 19:52:40.420899 3366 tablet_loader.cc:96] loaded metadata for tablet 7d2e58cc18b94f9ab036e3376920351c (table default.Table_d090d4e81b7c45e49c707734cfeae659 [id=02eadd9157e44aaf861adbfd95b2fc9c])
I20250905 19:52:40.421772 3366 tablet_loader.cc:96] loaded metadata for tablet 7e8bfe047a6e4cc087eea75244da5b41 (table default.Table_4e6468a3541a49c6841da695333625ab [id=3b8405e986104e499f8399d62c1c5646])
I20250905 19:52:40.422636 3366 tablet_loader.cc:96] loaded metadata for tablet 7ef9c9c6f81c4daea9008ada454b0659 (table default.Table_24293fcc0f3244c4a016cbb7e189a24d [id=6e93f40cbcce44c2b92d8b165bd57499])
I20250905 19:52:40.423110 3366 tablet_loader.cc:96] loaded metadata for tablet 7f34bdf651d1412690039dc6ea1095d0 (table default.Table_f96871d807144eeebd6dd5559a55303e [id=2e34d8d56af24da29eccb8e83c62d967])
I20250905 19:52:40.423568 3366 tablet_loader.cc:96] loaded metadata for tablet 8092952d755848ccaba3a2ad94b66dad (table default.Table_7ddc30e3c74645d6ac2b87b49d6dc53e [id=cf3dba91980e471497c7ac515106660f])
I20250905 19:52:40.424088 1169 tablet_bootstrap.cc:654] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.424958 3366 tablet_loader.cc:96] loaded metadata for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_084f2c1b9d144e38af5948cfd007d374 [id=7b0316448a484002a5cc2e3db8d4cdf4])
I20250905 19:52:40.425386 3366 tablet_loader.cc:96] loaded metadata for tablet 862214a31059407baefef8a390aaea69 (table default.Table_572ff86c294b4e0faaed684f96c62536 [id=866f5c7aa464474fb1428233e77b7482])
I20250905 19:52:40.425750 3366 tablet_loader.cc:96] loaded metadata for tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_f6f360749f78488fa03d88b2cfad1817 [id=a5ecc77067244a09b1b8b3cea3ed44b7])
I20250905 19:52:40.426708 3366 tablet_loader.cc:96] loaded metadata for tablet 8e0c478efe044ad0867e47f49374a66b (table default.Table_d76eee5aa7234b76a0f3ac551230b3ab [id=32e1eddadceb43d88212d5f7d3007c5c])
I20250905 19:52:40.427507 3274 consensus_queue.cc:1035] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.427600 3366 tablet_loader.cc:96] loaded metadata for tablet 8ee9e38dc3574c1ba9190e06557423f6 (table default.Table_0b1ff79095aa492b843efd3cc222a0db [id=31e2a02f7bcd4935ae597b69ed2c330e])
I20250905 19:52:40.427994 3366 tablet_loader.cc:96] loaded metadata for tablet 8faf429d811347a88880e8198daa59cc (table default.Table_bac4d4565a874689b77c9fa8009f96b4 [id=8f707442ad794cbfaec7aeee92c3be0c])
I20250905 19:52:40.428876 3366 tablet_loader.cc:96] loaded metadata for tablet 91396422d0bb468395db04aa80262882 (table default.Table_c79e0b36b0fd47f28d4d02f9a7060cfb [id=4654c1f365104bc99d329be12f5d6fad])
I20250905 19:52:40.429761 3366 tablet_loader.cc:96] loaded metadata for tablet 923ad8e40a07409e8f6a0152a0b2ef75 (table default.Table_650ae210fee64c839acb6988b887b7c2 [id=b8c03bd2ab2b48c1ac002b4be69e1cc6])
I20250905 19:52:40.430660 3366 tablet_loader.cc:96] loaded metadata for tablet 95b32fff7d4745c79185a5d894ef3342 (table default.Table_b129adef13534aacba0c176d41a4fca2 [id=8cdf9eabe68249929157c3294ccc3b5c])
I20250905 19:52:40.431324 3366 tablet_loader.cc:96] loaded metadata for tablet 96aaaf6d94594861b545f831f77dcb55 (table default.Table_6e9009b9124c413eb5058a1f3294990a [id=78d19980c7cc4a79b1b1dccc0a7f674a])
I20250905 19:52:40.431743 3366 tablet_loader.cc:96] loaded metadata for tablet 97d949c3c24e40fe8fefd3a5758d57ea (table default.Table_65b1531f2ba945118e5fdcfb5603e704 [id=c25ebbbb9d494a88821547bb2f49c9c5])
I20250905 19:52:40.432371 3366 tablet_loader.cc:96] loaded metadata for tablet 98db7e2a9d6d4f4d984388a493295d72 (table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f])
I20250905 19:52:40.432753 3366 tablet_loader.cc:96] loaded metadata for tablet 98e4e6d274574c0580315dcc2323c3d6 (table default.Table_3e73d212c07541f5afb91c19b19745f3 [id=e509c1d97279402c894b47e21dd4fcd1])
I20250905 19:52:40.433399 3366 tablet_loader.cc:96] loaded metadata for tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3])
I20250905 19:52:40.433784 3366 tablet_loader.cc:96] loaded metadata for tablet 9b3acf17eab64913b5b33677196a5781 (table default.Table_9a006790dc8f41cf8cb1e4ffc3d77175 [id=ab8df3c08b7e417ab17415bd9ded404d])
I20250905 19:52:40.434142 3366 tablet_loader.cc:96] loaded metadata for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b])
I20250905 19:52:40.434912 3366 tablet_loader.cc:96] loaded metadata for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda (table default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e])
I20250905 19:52:40.436002 3366 tablet_loader.cc:96] loaded metadata for tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1])
I20250905 19:52:40.436589 3366 tablet_loader.cc:96] loaded metadata for tablet 9f732ae0627c4da5b1691cde9b4d5562 (table default.Table_d24bc38d75eb490880b8769aa303daeb [id=fe9667d9d9cf4b0e99651b827e50e4ac])
I20250905 19:52:40.437156 3366 tablet_loader.cc:96] loaded metadata for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77])
I20250905 19:52:40.438225 3366 tablet_loader.cc:96] loaded metadata for tablet a311f9b943d143a9a52d7771ce7186d7 (table default.Table_9c9640e097254520bd3c753e09ca7733 [id=f1887695902842aaad9da95a35f248b4])
I20250905 19:52:40.438220 904 raft_consensus.cc:2466] T 7ef9c9c6f81c4daea9008ada454b0659 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.438743 986 leader_election.cc:304] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.438934 3366 tablet_loader.cc:96] loaded metadata for tablet a3a50526bb604a218649b24cbf65a65e (table default.Table_fb47213914d74250866aa0cb93a06647 [id=7d87c49eea0045e7a45de3237fa5a716])
I20250905 19:52:40.439278 3274 raft_consensus.cc:2802] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.439447 3274 raft_consensus.cc:695] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.439671 3366 tablet_loader.cc:96] loaded metadata for tablet a551e641a19f4d3f924fb176c4a25ca7 (table default.Table_cd6dae29e54e4735a5252b0638fcf5b7 [id=134e34dafa0649068577f7c46f83c1ba])
I20250905 19:52:40.440706 1169 tablet_bootstrap.cc:492] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.440856 1169 ts_tablet_manager.cc:1397] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.001s
I20250905 19:52:40.439839 3274 consensus_queue.cc:237] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.441095 3366 tablet_loader.cc:96] loaded metadata for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913])
I20250905 19:52:40.441725 1169 raft_consensus.cc:357] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.441902 1169 raft_consensus.cc:383] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.441967 1169 raft_consensus.cc:738] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.442209 1169 consensus_queue.cc:260] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.442796 3366 tablet_loader.cc:96] loaded metadata for tablet ac2e51a9071c4e1dae9e147687291175 (table default.Table_0aa187e069454d9abe28d7400d6067e7 [id=7a7fddc36ba34622a37aab225f4848ef])
I20250905 19:52:40.442860 1169 ts_tablet_manager.cc:1428] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:40.443169 1169 tablet_bootstrap.cc:492] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.443604 3366 tablet_loader.cc:96] loaded metadata for tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf])
I20250905 19:52:40.444077 3366 tablet_loader.cc:96] loaded metadata for tablet ad956757086b4d728698679c63c9b7ee (table default.Table_c1b824c37b3b4413b20c5b2172059490 [id=47bee8087f0943199f82be1345be78fd])
I20250905 19:52:40.444551 3366 tablet_loader.cc:96] loaded metadata for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e])
I20250905 19:52:40.445050 1169 tablet_bootstrap.cc:654] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.445273 3366 tablet_loader.cc:96] loaded metadata for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd])
I20250905 19:52:40.445762 3366 tablet_loader.cc:96] loaded metadata for tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75])
I20250905 19:52:40.446693 764 raft_consensus.cc:3058] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.447119 3366 tablet_loader.cc:96] loaded metadata for tablet b1f46261822442e0a961d2d63bdb35c8 (table default.Table_e32bc15eee6447be8dcf60f46cc58e12 [id=f2d0b16decef4ca8901260154a4fa801])
I20250905 19:52:40.447566 3366 tablet_loader.cc:96] loaded metadata for tablet b2c7906a718c46d9a76587a70d96a1f9 (table default.Table_fe3dee104cc640359210483bc31532cf [id=99384aac4e624d6db630b249c3ba2543])
I20250905 19:52:40.448014 3366 tablet_loader.cc:96] loaded metadata for tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0])
I20250905 19:52:40.449855 3299 consensus_queue.cc:1035] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.451100 3366 tablet_loader.cc:96] loaded metadata for tablet bb2b5ce70b784c13aa64053b2d1fe541 (table default.Table_57684ee2f11b44129e0b3d35029151fc [id=d636c310314a405e8d4f99e56d94f44c])
I20250905 19:52:40.451752 3366 tablet_loader.cc:96] loaded metadata for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498])
I20250905 19:52:40.452230 3366 tablet_loader.cc:96] loaded metadata for tablet be1315824fd749049a62af3e6a113b9a (table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f])
I20250905 19:52:40.452644 3366 tablet_loader.cc:96] loaded metadata for tablet be73873da994402e84fff6aa2ded7525 (table default.Table_6999a7918e524a0293956f96df1caa80 [id=5190c11e4b1142909f87e75ce899c422])
I20250905 19:52:40.453857 3366 tablet_loader.cc:96] loaded metadata for tablet c3b2fcc62ff0415f8073edb79b844600 (table default.Table_e6b9a48ec5ca435a9617f9cb52cd6ea0 [id=beb46dbcc0634881b6c69db37796ce3c])
I20250905 19:52:40.454532 3366 tablet_loader.cc:96] loaded metadata for tablet c4b31443b0f840749e12435e197a204a (table default.Table_890379e1838248739aa892ccc074b89a [id=500af28163d54d1d9e96c7b266592f4f])
I20250905 19:52:40.454944 3366 tablet_loader.cc:96] loaded metadata for tablet c69a41cee5ff4f8f9bccaa513c096118 (table default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d])
I20250905 19:52:40.455507 3366 tablet_loader.cc:96] loaded metadata for tablet c77dd068ea5c495ca8aa0e7a6823d792 (table default.Table_78bb4fad681f4443a890340c78ec736d [id=b3f6e906472a4471be3c9bdd0a25b96d])
I20250905 19:52:40.456425 3366 tablet_loader.cc:96] loaded metadata for tablet c870e689af254fb0acb15330577e5c42 (table default.Table_01c7456a16814955b2c2f3890ecb0d01 [id=525ecc2d31fc4e999f35b7514ea01c52])
I20250905 19:52:40.456409 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.456921 984 leader_election.cc:343] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.457090 3366 tablet_loader.cc:96] loaded metadata for tablet c8e53152ada84954b12bbfe9d1fe3c16 (table default.Table_af420aac7cc249339982b49a7f3942a2 [id=33f28b1e3d8f4fd0b036f269410dca92])
I20250905 19:52:40.458076 3366 tablet_loader.cc:96] loaded metadata for tablet ca646f7c162e41469f283657e72823b4 (table default.Table_015ff1997237469aa213edd41b3b6ea4 [id=737e1ec3455147b9badcb731f2fc7fc9])
I20250905 19:52:40.458931 3366 tablet_loader.cc:96] loaded metadata for tablet cd5312852fe345d98039aeea241b7fe8 (table default.Table_3ee66cc5b45647de889e10775569ae66 [id=ac18f9180d534577b99f576659e3615d])
I20250905 19:52:40.459589 3366 tablet_loader.cc:96] loaded metadata for tablet cff0e7cc7c814627ad8964fa6a3b1576 (table default.Table_487dad5063df425c8b4a44d7f8adbd39 [id=866aff163c794895abdd7632b46a252b])
I20250905 19:52:40.460022 3366 tablet_loader.cc:96] loaded metadata for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a])
I20250905 19:52:40.460774 3366 tablet_loader.cc:96] loaded metadata for tablet d1a11777cf2d4ea9a5cdf792f73aff76 (table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d])
I20250905 19:52:40.461056 1169 tablet_bootstrap.cc:492] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.461203 1169 ts_tablet_manager.cc:1397] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.018s user 0.000s sys 0.003s
I20250905 19:52:40.462062 1169 raft_consensus.cc:357] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.462265 1169 raft_consensus.cc:383] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.462333 1169 raft_consensus.cc:738] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.462571 1169 consensus_queue.cc:260] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.463133 1169 ts_tablet_manager.cc:1428] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:40.463193 3366 tablet_loader.cc:96] loaded metadata for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55])
I20250905 19:52:40.463446 1169 tablet_bootstrap.cc:492] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.463754 3366 tablet_loader.cc:96] loaded metadata for tablet d59da39e7bcf497a93fb9174328c1814 (table default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1])
I20250905 19:52:40.464217 3366 tablet_loader.cc:96] loaded metadata for tablet d6b44c00d3b54af6b0752575e6898839 (table default.Table_80c2578ee60e4142b0b77d1beba7e122 [id=142651eb40f74278862bb9858d94999c])
I20250905 19:52:40.464727 3366 tablet_loader.cc:96] loaded metadata for tablet d709e2b4f10149a1818c1e1431b95937 (table default.Table_19682dbce0bc4480b82d39235ba7f55b [id=bff29fe995d34f32ad9780ad5591a8b2])
I20250905 19:52:40.465292 1169 tablet_bootstrap.cc:654] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.465596 3366 tablet_loader.cc:96] loaded metadata for tablet d7155cf5abba4e2cbdce583406f94fad (table default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816])
I20250905 19:52:40.466277 3366 tablet_loader.cc:96] loaded metadata for tablet db098a883a61474fb06cb166041ee3e4 (table default.Table_dccb2b41c5664fcda055f919170fe6cd [id=0053f42013184bbf90b4ad602b4de9b2])
I20250905 19:52:40.466725 3366 tablet_loader.cc:96] loaded metadata for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (table default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786])
I20250905 19:52:40.467398 3366 tablet_loader.cc:96] loaded metadata for tablet dcf448f9abc348df8d2d2501420d8f45 (table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727])
I20250905 19:52:40.467810 3366 tablet_loader.cc:96] loaded metadata for tablet de19c897d3314fecb17cedb3370eed93 (table default.Table_906ecce79e3a438c82eb0f15671e1d07 [id=d339af77e682450cb04506c0f0a09bdb])
I20250905 19:52:40.468268 3366 tablet_loader.cc:96] loaded metadata for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b])
I20250905 19:52:40.468901 3366 tablet_loader.cc:96] loaded metadata for tablet e3cb1b098f42441594e28aeb1a9dbbf1 (table default.Table_e02928704120437bbdda2c9819761a85 [id=c08a19b2c2384f52b2f91b47ef74e13f])
I20250905 19:52:40.469353 3366 tablet_loader.cc:96] loaded metadata for tablet e44e88a0df444fe5b156a8fdf3199507 (table default.Table_f34aac6beca44bc3bf4326aa500aaf21 [id=c96257f6e492466b88ad6da8f4abeb84])
I20250905 19:52:40.469784 3366 tablet_loader.cc:96] loaded metadata for tablet e476304b492f4bd69d5ea0f109263517 (table default.Table_3e897c08965d4265b2a5937b15f5c715 [id=48d7bc33c38d44fc977294a75d895a15])
I20250905 19:52:40.470209 3366 tablet_loader.cc:96] loaded metadata for tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4])
I20250905 19:52:40.470937 3366 tablet_loader.cc:96] loaded metadata for tablet e5f9e807244c4e0eb1f6dd1fc2acb9c1 (table default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f [id=22c178eea85f4048b10fb58b135663d6])
I20250905 19:52:40.471534 3366 tablet_loader.cc:96] loaded metadata for tablet e5fc2c2ba49444af939f5a975165362b (table default.Table_6d527034680144f190c70d035f183b04 [id=0288ac8944474683ba7b18358be1849b])
I20250905 19:52:40.472018 3366 tablet_loader.cc:96] loaded metadata for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab])
I20250905 19:52:40.472483 3366 tablet_loader.cc:96] loaded metadata for tablet e67ed7e9e5724de389504701584bd2b6 (table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923])
W20250905 19:52:40.472247 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 96576us
I20250905 19:52:40.473757 3366 tablet_loader.cc:96] loaded metadata for tablet e7086c5bfa4644d08e4f55231b85acf4 (table default.Table_71d2ee92e2ba427fb82bb9120ea13ad5 [id=f09f8318d7a84f22b8d82f13614d5d39])
I20250905 19:52:40.474293 3366 tablet_loader.cc:96] loaded metadata for tablet e77bdf9df6eb4470a3d22e96da952718 (table default.Table_96c633b175ac41309951506efe002443 [id=ae92837e2ac94949aac8822f084c0a97])
I20250905 19:52:40.474694 3366 tablet_loader.cc:96] loaded metadata for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151])
W20250905 19:52:40.475064 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 80276.2us
I20250905 19:52:40.475265 1170 raft_consensus.cc:357] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.475440 1170 raft_consensus.cc:383] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.475485 1170 raft_consensus.cc:738] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.475697 1170 consensus_queue.cc:260] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.476200 1170 ts_tablet_manager.cc:1428] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20250905 19:52:40.476464 1170 tablet_bootstrap.cc:492] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.478324 1170 tablet_bootstrap.cc:654] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.482957 3366 tablet_loader.cc:96] loaded metadata for tablet efb70327f55844879f89449a7116010f (table default.Table_d7e9ca2342f94ec49cb959b60816c1d3 [id=ff5964a447094f6ea231fb91971d83e7])
I20250905 19:52:40.483880 712 leader_election.cc:304] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:40.484398 3346 raft_consensus.cc:2747] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.485147 3366 tablet_loader.cc:96] loaded metadata for tablet efceaf3c7bda44ddabd5d38130af3dd2 (table default.Table_dc9077e09b234b828d9df52479655f50 [id=e266d1ebb88f4e2e9572f6c3ceaa8530])
I20250905 19:52:40.486133 3366 tablet_loader.cc:96] loaded metadata for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df])
I20250905 19:52:40.487635 3366 tablet_loader.cc:96] loaded metadata for tablet f2d929ec75d34cddad5286d8e37fcefb (table default.Table_1e1fa3461d184e1c82bf5ce46b0a0a84 [id=79afa9cf443b4a8db43032bb47c2c68b])
I20250905 19:52:40.488631 3366 tablet_loader.cc:96] loaded metadata for tablet f34f0b29d6104cac8ab35309fd1edae8 (table default.Table_b543aba1b13749578be5d9e1dd312c0d [id=93431c79f59f483f9f9523eec2aada09])
I20250905 19:52:40.490024 3366 tablet_loader.cc:96] loaded metadata for tablet f6f29ac5e68d4d8da8f3496fefe695ec (table default.Table_4fe8c2a1caf340069ff229767f87e772 [id=1196f4fec5bc41109112409f35a92cb3])
I20250905 19:52:40.492048 3366 tablet_loader.cc:96] loaded metadata for tablet fa5d34f9c0ee4d55ade61b30b0f4c293 (table default.Table_b740696e409f49ba9fc25fb046f7b6e8 [id=98f1d36f85754ac8aefa8743d9859ff9])
W20250905 19:52:40.494630 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 87325.8us
I20250905 19:52:40.494979 3282 raft_consensus.cc:491] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4eb3af79b01d4835bf6bf7b26bf016f5)
I20250905 19:52:40.495164 3282 raft_consensus.cc:513] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.495808 3282 leader_election.cc:290] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.497650 1170 tablet_bootstrap.cc:492] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.497845 1170 ts_tablet_manager.cc:1397] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.002s
I20250905 19:52:40.499111 1170 raft_consensus.cc:357] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.499423 1170 raft_consensus.cc:383] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.499517 1170 raft_consensus.cc:738] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.499835 1170 consensus_queue.cc:260] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.500684 1170 ts_tablet_manager.cc:1428] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:40.501363 1170 tablet_bootstrap.cc:492] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.505749 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.505951 907 raft_consensus.cc:2466] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:40.507616 712 leader_election.cc:304] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:40.507994 3282 raft_consensus.cc:2802] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:40.508072 3282 raft_consensus.cc:491] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4eb3af79b01d4835bf6bf7b26bf016f5)
I20250905 19:52:40.508139 3282 raft_consensus.cc:3058] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:40.510754 3282 raft_consensus.cc:513] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.511381 1169 tablet_bootstrap.cc:492] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.511507 1169 ts_tablet_manager.cc:1397] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.048s user 0.006s sys 0.006s
I20250905 19:52:40.511747 3282 leader_election.cc:290] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.512439 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.512609 907 raft_consensus.cc:3058] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:40.513238 3299 consensus_queue.cc:1035] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.515234 907 raft_consensus.cc:2466] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 2.
I20250905 19:52:40.515697 904 raft_consensus.cc:1238] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Rejecting Update request from peer 4eb3af79b01d4835bf6bf7b26bf016f5 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250905 19:52:40.516294 1169 raft_consensus.cc:357] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.516467 1169 raft_consensus.cc:383] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.516391 3299 consensus_queue.cc:1046] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.516521 1169 raft_consensus.cc:738] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.516726 2931 raft_consensus.cc:3053] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:52:40.516808 2931 raft_consensus.cc:738] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.516747 1169 consensus_queue.cc:260] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.517063 2931 consensus_queue.cc:260] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.517300 1169 ts_tablet_manager.cc:1428] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:40.517351 2931 raft_consensus.cc:3058] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:40.517606 712 leader_election.cc:304] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:40.517875 1169 tablet_bootstrap.cc:492] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.517907 3282 raft_consensus.cc:2802] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:40.518172 3282 raft_consensus.cc:695] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:40.518455 3282 consensus_queue.cc:237] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.522318 1169 tablet_bootstrap.cc:654] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.522856 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.523030 1049 raft_consensus.cc:2466] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 2.
I20250905 19:52:40.524003 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "f34f0b29d6104cac8ab35309fd1edae8" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:40.524874 3366 tablet_loader.cc:96] loaded metadata for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7])
I20250905 19:52:40.525398 3366 tablet_loader.cc:96] loaded metadata for tablet fcb9bcc5e8634211a72cc05e936c71e7 (table default.Table_5bf03647f71d46a8b3d3f0b5bc385154 [id=e19dba4ecf6d4c22965720b89e7c45d4])
I20250905 19:52:40.526176 3366 tablet_loader.cc:96] loaded metadata for tablet fe00969eb9654976a0efeac2fa7c91d3 (table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4])
I20250905 19:52:40.526983 1048 raft_consensus.cc:2466] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 2.
I20250905 19:52:40.533789 3366 tablet_loader.cc:96] loaded metadata for tablet ff2f289c293f41379e4af470d747860c (table default.Table_20b48d10bd25499c9543b27963456363 [id=30bde1454edc43dbae0ef83402b0fdb5])
W20250905 19:52:40.534686 3366 catalog_manager.cc:1478] Time spent T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Loading table and tablet metadata into memory: real 1.269s user 0.199s sys 0.054s
I20250905 19:52:40.535797 3366 catalog_manager.cc:1486] Initializing Kudu cluster ID...
I20250905 19:52:40.547827 1169 tablet_bootstrap.cc:492] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.548038 1169 ts_tablet_manager.cc:1397] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.030s user 0.005s sys 0.002s
I20250905 19:52:40.549120 1169 raft_consensus.cc:357] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.549296 1169 raft_consensus.cc:383] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.549351 1169 raft_consensus.cc:738] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.549588 1169 consensus_queue.cc:260] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.550383 1169 ts_tablet_manager.cc:1428] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:40.550658 1169 tablet_bootstrap.cc:492] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.553932 1169 tablet_bootstrap.cc:654] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.557446 3366 catalog_manager.cc:1261] Loaded cluster ID: 3f57394f817640c5aa082abf33221723
I20250905 19:52:40.557617 3366 catalog_manager.cc:1497] Initializing Kudu internal certificate authority...
I20250905 19:52:40.570533 1170 tablet_bootstrap.cc:654] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.575587 1170 tablet_bootstrap.cc:492] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.575752 1170 ts_tablet_manager.cc:1397] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.075s user 0.013s sys 0.006s
I20250905 19:52:40.576038 3366 catalog_manager.cc:1506] Loading token signing keys...
I20250905 19:52:40.576447 1170 raft_consensus.cc:357] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.576620 1170 raft_consensus.cc:383] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.576697 1170 raft_consensus.cc:738] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.576884 1170 consensus_queue.cc:260] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.577373 1170 ts_tablet_manager.cc:1428] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:40.577621 1170 tablet_bootstrap.cc:492] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.579006 1170 tablet_bootstrap.cc:654] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.582592 3274 raft_consensus.cc:491] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.582728 3274 raft_consensus.cc:513] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.583889 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bd731a303a545f492b19f774520582c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.584141 904 raft_consensus.cc:2466] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.584544 986 leader_election.cc:304] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.584829 3299 raft_consensus.cc:2802] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.584913 3299 raft_consensus.cc:491] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.584868 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bd731a303a545f492b19f774520582c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.584986 3299 raft_consensus.cc:3058] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.585047 767 raft_consensus.cc:2466] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.587677 3299 raft_consensus.cc:513] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.591831 1170 tablet_bootstrap.cc:492] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.592010 1170 ts_tablet_manager.cc:1397] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.004s sys 0.002s
I20250905 19:52:40.592675 1170 raft_consensus.cc:357] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.592831 1170 raft_consensus.cc:383] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.592893 1170 raft_consensus.cc:738] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.593109 3274 leader_election.cc:290] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.593124 1170 consensus_queue.cc:260] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.593753 1170 ts_tablet_manager.cc:1428] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:40.594393 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bd731a303a545f492b19f774520582c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.594565 904 raft_consensus.cc:3058] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.597268 904 raft_consensus.cc:2466] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.600870 986 leader_election.cc:304] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.601226 3297 raft_consensus.cc:2802] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.601342 3297 raft_consensus.cc:695] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.601574 3297 consensus_queue.cc:237] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.605149 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1bd731a303a545f492b19f774520582c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.605340 767 raft_consensus.cc:3058] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.605795 1169 tablet_bootstrap.cc:492] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.605911 1169 ts_tablet_manager.cc:1397] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.055s user 0.004s sys 0.000s
I20250905 19:52:40.606621 1169 raft_consensus.cc:357] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.606830 1169 raft_consensus.cc:383] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.606876 1169 raft_consensus.cc:738] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.607040 1169 consensus_queue.cc:260] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.607376 1169 ts_tablet_manager.cc:1428] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.001s sys 0.000s
I20250905 19:52:40.607816 1170 tablet_bootstrap.cc:492] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.609349 1170 tablet_bootstrap.cc:654] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.609558 1169 tablet_bootstrap.cc:492] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.611204 1169 tablet_bootstrap.cc:654] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.611336 767 raft_consensus.cc:2466] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.614917 3299 leader_election.cc:290] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.626495 1169 tablet_bootstrap.cc:492] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.626641 1169 ts_tablet_manager.cc:1397] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20250905 19:52:40.627368 1169 raft_consensus.cc:357] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.627519 1169 raft_consensus.cc:383] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.627559 1169 raft_consensus.cc:738] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.627703 1169 consensus_queue.cc:260] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.628187 1169 ts_tablet_manager.cc:1428] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.002s
I20250905 19:52:40.628453 1169 tablet_bootstrap.cc:492] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.630167 1169 tablet_bootstrap.cc:654] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.638726 1169 tablet_bootstrap.cc:492] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.638863 1169 ts_tablet_manager.cc:1397] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.000s
I20250905 19:52:40.639518 1169 raft_consensus.cc:357] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.639680 1169 raft_consensus.cc:383] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.639729 1169 raft_consensus.cc:738] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.639909 1169 consensus_queue.cc:260] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.640440 1169 ts_tablet_manager.cc:1428] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20250905 19:52:40.640736 1169 tablet_bootstrap.cc:492] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.642452 1169 tablet_bootstrap.cc:654] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.646379 1170 tablet_bootstrap.cc:492] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.646595 1170 ts_tablet_manager.cc:1397] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.039s user 0.002s sys 0.007s
I20250905 19:52:40.647285 1170 raft_consensus.cc:357] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.647507 1170 raft_consensus.cc:383] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.647599 1170 raft_consensus.cc:738] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.647938 1170 consensus_queue.cc:260] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.648789 1170 ts_tablet_manager.cc:1428] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:40.649097 1170 tablet_bootstrap.cc:492] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.650810 1170 tablet_bootstrap.cc:654] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.665058 1169 tablet_bootstrap.cc:492] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.665205 1169 ts_tablet_manager.cc:1397] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.002s
I20250905 19:52:40.665889 1169 raft_consensus.cc:357] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.666083 1169 raft_consensus.cc:383] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.666133 1169 raft_consensus.cc:738] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.666312 1169 consensus_queue.cc:260] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.666766 1169 ts_tablet_manager.cc:1428] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:40.667068 1169 tablet_bootstrap.cc:492] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.668823 1169 tablet_bootstrap.cc:654] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.673223 3282 raft_consensus.cc:491] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.673359 3282 raft_consensus.cc:513] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.673941 3282 leader_election.cc:290] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.674474 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.674677 904 raft_consensus.cc:2391] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:40.674918 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "03cf686811e041e7a001669b3d86e28a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.675452 712 leader_election.cc:304] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:40.675916 3282 raft_consensus.cc:3058] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.678493 3282 raft_consensus.cc:2747] T 03cf686811e041e7a001669b3d86e28a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.678936 3297 raft_consensus.cc:491] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.680697 3297 raft_consensus.cc:513] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.681263 1170 tablet_bootstrap.cc:492] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.681648 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.681870 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.682057 904 raft_consensus.cc:2466] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
W20250905 19:52:40.682185 984 leader_election.cc:343] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.682420 1170 ts_tablet_manager.cc:1397] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.001s
I20250905 19:52:40.682524 986 leader_election.cc:304] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.682855 3299 raft_consensus.cc:2802] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.682927 3299 raft_consensus.cc:491] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.683000 3299 raft_consensus.cc:3058] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.683864 767 raft_consensus.cc:3058] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.685639 3299 raft_consensus.cc:513] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.687052 3299 leader_election.cc:290] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.687294 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.687351 1170 raft_consensus.cc:357] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.687461 904 raft_consensus.cc:3058] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.687494 1170 raft_consensus.cc:383] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.687547 1170 raft_consensus.cc:738] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.687836 1170 consensus_queue.cc:260] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.688748 1170 ts_tablet_manager.cc:1428] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250905 19:52:40.689005 1170 tablet_bootstrap.cc:492] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.690219 904 raft_consensus.cc:2466] T e3cb1b098f42441594e28aeb1a9dbbf1 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.690747 1170 tablet_bootstrap.cc:654] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.690795 986 leader_election.cc:304] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.692437 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.692679 3274 raft_consensus.cc:2802] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.692786 3274 raft_consensus.cc:695] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.693055 3274 consensus_queue.cc:237] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:40.693142 984 leader_election.cc:343] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.697058 3297 leader_election.cc:290] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.699199 3282 raft_consensus.cc:491] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.699365 3282 raft_consensus.cc:513] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.699630 3366 catalog_manager.cc:5966] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Loaded TSK: 0
I20250905 19:52:40.700435 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.700706 904 raft_consensus.cc:2391] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:40.702272 3366 catalog_manager.cc:1516] Initializing in-progress tserver states...
I20250905 19:52:40.703213 3282 leader_election.cc:290] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.703780 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fbba1ad28a614b06afe865c478901a32" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.704427 711 leader_election.cc:304] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:40.704790 3282 raft_consensus.cc:3058] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.707667 3282 raft_consensus.cc:2747] T fbba1ad28a614b06afe865c478901a32 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.709687 1169 tablet_bootstrap.cc:492] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.709844 1169 ts_tablet_manager.cc:1397] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.043s user 0.004s sys 0.008s
I20250905 19:52:40.713161 1169 raft_consensus.cc:357] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.713366 1169 raft_consensus.cc:383] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.713420 1169 raft_consensus.cc:738] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.713608 1169 consensus_queue.cc:260] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.714229 1169 ts_tablet_manager.cc:1428] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:40.714516 1169 tablet_bootstrap.cc:492] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.716224 1169 tablet_bootstrap.cc:654] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.716959 3297 raft_consensus.cc:491] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.717094 3297 raft_consensus.cc:513] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.717686 3297 leader_election.cc:290] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.719296 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.719473 904 raft_consensus.cc:2466] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.720049 986 leader_election.cc:304] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.720388 3297 raft_consensus.cc:2802] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.720463 3297 raft_consensus.cc:491] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.720525 3297 raft_consensus.cc:3058] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.722616 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.726384 984 leader_election.cc:343] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.726709 3297 raft_consensus.cc:513] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.731413 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.731602 904 raft_consensus.cc:3058] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.732616 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.733105 984 leader_election.cc:343] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.734386 904 raft_consensus.cc:2466] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.735023 986 leader_election.cc:304] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.735162 2931 raft_consensus.cc:491] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.735291 3299 raft_consensus.cc:2802] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.735399 3299 raft_consensus.cc:695] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.735610 3299 consensus_queue.cc:237] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.735302 2931 raft_consensus.cc:513] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.736380 2931 leader_election.cc:290] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.737010 764 raft_consensus.cc:3058] T 74eb2632a6414e52b9ad67f53c182679 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.740545 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff0e7cc7c814627ad8964fa6a3b1576" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.740989 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff0e7cc7c814627ad8964fa6a3b1576" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.741170 904 raft_consensus.cc:2466] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.741604 986 leader_election.cc:304] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:40.741716 984 leader_election.cc:343] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.741941 3151 raft_consensus.cc:2802] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.742015 3151 raft_consensus.cc:491] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.742076 3151 raft_consensus.cc:3058] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.742120 2496 consensus_queue.cc:1035] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.744702 3151 raft_consensus.cc:513] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.745235 3151 leader_election.cc:290] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.745923 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff0e7cc7c814627ad8964fa6a3b1576" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.746080 907 raft_consensus.cc:3058] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.748669 907 raft_consensus.cc:2466] T cff0e7cc7c814627ad8964fa6a3b1576 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.749274 986 leader_election.cc:304] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:40.752436 3151 raft_consensus.cc:491] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.752497 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cff0e7cc7c814627ad8964fa6a3b1576" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.752578 3151 raft_consensus.cc:513] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.753121 3151 leader_election.cc:290] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.753871 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:40.756459 3297 leader_election.cc:290] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.758001 1170 tablet_bootstrap.cc:492] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.758152 1170 ts_tablet_manager.cc:1397] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.069s user 0.003s sys 0.000s
I20250905 19:52:40.758935 1170 raft_consensus.cc:357] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.759123 1170 raft_consensus.cc:383] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.759179 1170 raft_consensus.cc:738] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.763571 1170 consensus_queue.cc:260] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.764377 1170 ts_tablet_manager.cc:1428] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20250905 19:52:40.764678 1170 tablet_bootstrap.cc:492] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.767690 1170 tablet_bootstrap.cc:654] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:40.789461 984 leader_election.cc:343] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:40.790346 984 leader_election.cc:343] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.791025 769 raft_consensus.cc:3058] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.791628 3274 raft_consensus.cc:491] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.791755 3274 raft_consensus.cc:513] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.792397 3274 leader_election.cc:290] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.792969 1169 tablet_bootstrap.cc:492] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:40.793094 1169 ts_tablet_manager.cc:1397] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.079s user 0.017s sys 0.013s
I20250905 19:52:40.793499 2931 consensus_queue.cc:1035] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.793816 1169 raft_consensus.cc:357] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.794704 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.795336 984 leader_election.cc:343] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.803615 3151 consensus_queue.cc:1035] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.816201 2405 master_service.cc:948] ReplaceTablet: received request to replace tablet 5fdc9d9815864aa7bf5ca9a5c4485390 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:40.818856 2399 master_service.cc:948] ReplaceTablet: received request to replace tablet 421b28d047124222ae9cbc603ec22cee from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:40.820125 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_ce82d3ecb4894b59aa7e82b457011d4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:40.822252 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_ce82d3ecb4894b59aa7e82b457011d4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:40.824066 1170 tablet_bootstrap.cc:492] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.824102 2403 master_service.cc:948] ReplaceTablet: received request to replace tablet 9a81376d4c0b47a9817deae4d34ffb47 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:40.824200 1170 ts_tablet_manager.cc:1397] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.060s user 0.004s sys 0.010s
I20250905 19:52:40.824987 1170 raft_consensus.cc:357] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.825172 1170 raft_consensus.cc:383] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.825222 1170 raft_consensus.cc:738] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.825407 1170 consensus_queue.cc:260] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.825381 2392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_124862b3c64045b4bc73ab2cb73400fa"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:40.825881 1170 ts_tablet_manager.cc:1428] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
W20250905 19:52:40.825978 2392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_124862b3c64045b4bc73ab2cb73400fa in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:40.826165 1170 tablet_bootstrap.cc:492] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.827044 2373 master_service.cc:948] ReplaceTablet: received request to replace tablet 89a73e984cbb4268b563f1ca01f4c65d from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:40.828351 1170 tablet_bootstrap.cc:654] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.828405 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_f779bfa94c9a4420ab2ec42650434ac1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:40.828977 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f779bfa94c9a4420ab2ec42650434ac1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:40.830144 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_8b15bc378227402a9b804f9e9f13c83e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:40.799460 3274 raft_consensus.cc:2802] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
W20250905 19:52:40.830698 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b15bc378227402a9b804f9e9f13c83e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:40.830705 3274 raft_consensus.cc:695] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.831003 3274 consensus_queue.cc:237] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.832219 2340 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_442ce59f4a4c4f1aac9b1c83d7535fbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:40.832728 2340 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_442ce59f4a4c4f1aac9b1c83d7535fbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:40.799811 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 50566.7us
I20250905 19:52:40.831863 2342 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_596eb1526ec44e39bc70a926d0ed1d0c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:40.835954 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:40.836066 2342 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_596eb1526ec44e39bc70a926d0ed1d0c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:40.836194 904 raft_consensus.cc:2466] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.838557 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.838727 904 raft_consensus.cc:2466] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.840677 3282 raft_consensus.cc:491] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.840806 3282 raft_consensus.cc:513] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.841696 3282 leader_election.cc:290] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:40.842075 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.842113 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e67ed7e9e5724de389504701584bd2b6" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:40.842271 904 raft_consensus.cc:2391] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:40.842665 711 leader_election.cc:304] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:40.843093 3282 raft_consensus.cc:3058] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.843405 986 leader_election.cc:304] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.844066 3151 raft_consensus.cc:491] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.844199 3151 raft_consensus.cc:513] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.844588 2496 raft_consensus.cc:2802] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.844657 2496 raft_consensus.cc:491] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.844724 2496 raft_consensus.cc:3058] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.844790 3151 leader_election.cc:290] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.845590 3282 raft_consensus.cc:2747] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:40.846889 986 leader_election.cc:304] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.847105 2931 raft_consensus.cc:2802] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.847178 2931 raft_consensus.cc:491] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.847232 2931 raft_consensus.cc:3058] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.847337 3297 consensus_queue.cc:1035] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.848115 3151 consensus_queue.cc:1035] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.848344 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.848866 984 leader_election.cc:343] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.849834 2931 raft_consensus.cc:513] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.850346 2931 leader_election.cc:290] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.850502 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.850661 904 raft_consensus.cc:2466] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.847237 2496 raft_consensus.cc:513] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.851481 2496 leader_election.cc:290] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.852059 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.854453 984 leader_election.cc:343] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.860327 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.863636 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.863827 906 raft_consensus.cc:3058] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.866485 906 raft_consensus.cc:2466] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.867904 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.868176 906 raft_consensus.cc:3058] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:40.870525 984 leader_election.cc:343] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.871124 906 raft_consensus.cc:2466] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.872256 1170 tablet_bootstrap.cc:492] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.872254 2496 consensus_queue.cc:1035] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.872391 1170 ts_tablet_manager.cc:1397] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.046s user 0.004s sys 0.004s
I20250905 19:52:40.872983 986 leader_election.cc:304] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.873317 1170 raft_consensus.cc:357] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.873490 1170 raft_consensus.cc:383] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.873545 1170 raft_consensus.cc:738] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.873729 1170 consensus_queue.cc:260] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.874048 2496 raft_consensus.cc:2802] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.874133 2496 raft_consensus.cc:491] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.874213 2496 raft_consensus.cc:3058] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.874492 1170 ts_tablet_manager.cc:1428] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:40.874745 1170 tablet_bootstrap.cc:492] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.874794 2931 consensus_queue.cc:1035] T a551e641a19f4d3f924fb176c4a25ca7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.876855 2496 raft_consensus.cc:513] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.877449 2496 leader_election.cc:290] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.877846 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
W20250905 19:52:40.878412 984 leader_election.cc:343] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.793960 1169 raft_consensus.cc:383] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.884033 1169 raft_consensus.cc:738] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.884456 1169 consensus_queue.cc:260] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.884907 2496 consensus_queue.cc:1035] T 7e8bfe047a6e4cc087eea75244da5b41 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.885229 986 leader_election.cc:304] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.885725 2496 raft_consensus.cc:2802] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.885841 2496 raft_consensus.cc:695] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.886060 2496 consensus_queue.cc:237] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.888947 986 leader_election.cc:304] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.889175 2931 raft_consensus.cc:2802] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.889302 2931 raft_consensus.cc:695] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.889537 2931 consensus_queue.cc:237] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.890240 1170 tablet_bootstrap.cc:654] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.892969 1169 ts_tablet_manager.cc:1428] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.100s user 0.000s sys 0.002s
I20250905 19:52:40.893162 3299 consensus_queue.cc:1035] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:40.894038 1169 tablet_bootstrap.cc:492] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:40.895745 3297 raft_consensus.cc:491] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.895867 3297 raft_consensus.cc:513] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.896447 3297 leader_election.cc:290] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.897367 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a5d38136c034b8497045176f749a98d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.897991 984 leader_election.cc:343] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.903712 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:40.903949 907 raft_consensus.cc:3058] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.904902 3297 raft_consensus.cc:491] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.905050 3297 raft_consensus.cc:513] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.904955 2397 catalog_manager.cc:5582] T 74eb2632a6414e52b9ad67f53c182679 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:40.905597 3297 leader_election.cc:290] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.905754 2397 catalog_manager.cc:5582] T 5aff2acc3f8245e1bca41f971f8e74ce P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:40.907207 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad956757086b4d728698679c63c9b7ee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.907887 984 leader_election.cc:343] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.911245 907 raft_consensus.cc:2466] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.913057 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a5d38136c034b8497045176f749a98d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.913374 906 raft_consensus.cc:2466] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.917363 1170 tablet_bootstrap.cc:492] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.917506 1170 ts_tablet_manager.cc:1397] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.043s user 0.001s sys 0.002s
I20250905 19:52:40.918217 1170 raft_consensus.cc:357] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.918387 1170 raft_consensus.cc:383] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.918437 1170 raft_consensus.cc:738] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.918731 3297 raft_consensus.cc:491] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.918642 1170 consensus_queue.cc:260] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.918975 1169 tablet_bootstrap.cc:654] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.919332 1170 ts_tablet_manager.cc:1428] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:40.919253 3297 raft_consensus.cc:513] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:40.919677 1170 tablet_bootstrap.cc:492] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.920192 3297 leader_election.cc:290] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.920933 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad956757086b4d728698679c63c9b7ee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:40.921121 906 raft_consensus.cc:2466] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:40.921589 986 leader_election.cc:304] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.921967 986 leader_election.cc:304] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.922325 986 leader_election.cc:304] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:40.922587 3299 raft_consensus.cc:2802] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.922698 3299 raft_consensus.cc:695] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.923004 3299 consensus_queue.cc:237] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.924463 3151 raft_consensus.cc:2802] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.924546 3151 raft_consensus.cc:491] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.924618 3151 raft_consensus.cc:3058] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.927179 3151 raft_consensus.cc:513] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.927695 3151 leader_election.cc:290] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.927832 3151 raft_consensus.cc:2802] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:40.928948 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a5d38136c034b8497045176f749a98d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:40.929154 770 raft_consensus.cc:3058] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.932951 770 raft_consensus.cc:2466] T 5a5d38136c034b8497045176f749a98d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:40.933866 984 leader_election.cc:304] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:40.934371 3297 raft_consensus.cc:2802] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:40.934535 3297 raft_consensus.cc:695] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:40.934835 3297 consensus_queue.cc:237] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.937120 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10fef848b9b746c5a0de6a8c3ca74f7a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:40.939401 984 leader_election.cc:343] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:40.927942 3151 raft_consensus.cc:491] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:40.945025 3151 raft_consensus.cc:3058] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:40.952193 1170 tablet_bootstrap.cc:654] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.959767 2405 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 5fdc9d9815864aa7bf5ca9a5c4485390
I20250905 19:52:40.961850 2405 catalog_manager.cc:6652] ReplaceTablet: tablet 5fdc9d9815864aa7bf5ca9a5c4485390 deleted and replaced by tablet 350197e134c549198771f4f8337d5dfc
I20250905 19:52:40.963299 1170 tablet_bootstrap.cc:492] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.963446 1170 ts_tablet_manager.cc:1397] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.044s user 0.004s sys 0.001s
I20250905 19:52:40.963433 3274 raft_consensus.cc:491] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:40.963598 3274 raft_consensus.cc:513] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.964332 3274 leader_election.cc:290] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.964305 1170 raft_consensus.cc:357] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.964485 1170 raft_consensus.cc:383] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.964541 1170 raft_consensus.cc:738] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.964742 1170 consensus_queue.cc:260] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.965245 1170 ts_tablet_manager.cc:1428] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:40.965504 1170 tablet_bootstrap.cc:492] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.959767 2403 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 9a81376d4c0b47a9817deae4d34ffb47
I20250905 19:52:40.966037 2403 catalog_manager.cc:6652] ReplaceTablet: tablet 9a81376d4c0b47a9817deae4d34ffb47 deleted and replaced by tablet 26905425077e4746bccb51431da706c9
I20250905 19:52:40.967410 1170 tablet_bootstrap.cc:654] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:40.986402 3151 raft_consensus.cc:513] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.987418 3151 leader_election.cc:290] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:40.987893 1170 tablet_bootstrap.cc:492] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:40.988083 1170 ts_tablet_manager.cc:1397] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.023s user 0.005s sys 0.001s
I20250905 19:52:40.989121 1170 raft_consensus.cc:357] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.989317 1170 raft_consensus.cc:383] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:40.989389 1170 raft_consensus.cc:738] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:40.989643 1170 consensus_queue.cc:260] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:40.990324 1170 ts_tablet_manager.cc:1428] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:40.990680 1170 tablet_bootstrap.cc:492] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:40.996197 2373 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 89a73e984cbb4268b563f1ca01f4c65d
I20250905 19:52:40.996852 2373 catalog_manager.cc:6652] ReplaceTablet: tablet 89a73e984cbb4268b563f1ca01f4c65d deleted and replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4
I20250905 19:52:40.998769 1170 tablet_bootstrap.cc:654] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.006026 3274 raft_consensus.cc:491] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.006234 3274 raft_consensus.cc:513] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.006959 3274 leader_election.cc:290] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.009872 2373 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_90a255f85d934941be4b07b8d27bdd8b" } modify_external_catalogs: true
I20250905 19:52:41.010134 2373 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_90a255f85d934941be4b07b8d27bdd8b" } modify_external_catalogs: true
I20250905 19:52:41.012310 2403 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_a0b7f5e829264dc5974e4d1039940a52"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.013190 2403 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0b7f5e829264dc5974e4d1039940a52 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.014464 2372 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_f6f360749f78488fa03d88b2cfad1817" } new_table_name: "default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952" modify_external_catalogs: true
I20250905 19:52:41.018182 2392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_870269fff0a1492e9a170850a3981347"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.018715 2392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_870269fff0a1492e9a170850a3981347 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.034122 1169 tablet_bootstrap.cc:492] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:41.034291 1169 ts_tablet_manager.cc:1397] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.140s user 0.009s sys 0.014s
I20250905 19:52:41.035036 1169 raft_consensus.cc:357] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.035249 1169 raft_consensus.cc:383] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.035308 1169 raft_consensus.cc:738] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.035497 1169 consensus_queue.cc:260] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.036034 1169 ts_tablet_manager.cc:1428] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:41.036350 1169 tablet_bootstrap.cc:492] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:41.037870 1169 tablet_bootstrap.cc:654] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:41.039614 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 117305us
I20250905 19:52:41.040215 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10fef848b9b746c5a0de6a8c3ca74f7a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.040405 904 raft_consensus.cc:2466] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.041708 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "5a5d38136c034b8497045176f749a98d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.041976 907 raft_consensus.cc:3058] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.045957 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.046173 904 raft_consensus.cc:2466] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.046938 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad956757086b4d728698679c63c9b7ee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.047108 904 raft_consensus.cc:3058] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.049892 904 raft_consensus.cc:2466] T ad956757086b4d728698679c63c9b7ee P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.051460 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.051746 904 raft_consensus.cc:2466] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.053908 2370 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_fd318ab664334e4bb9d57d696f6449d2" } modify_external_catalogs: true
I20250905 19:52:41.054114 2370 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_fd318ab664334e4bb9d57d696f6449d2" } modify_external_catalogs: true
I20250905 19:52:41.057425 2342 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_78bb4fad681f4443a890340c78ec736d" } new_table_name: "default.Table_e08447fc15f5401eb931185a79ad62c8" modify_external_catalogs: true
I20250905 19:52:41.057898 2399 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_1f75eecc8c8941a6aa9982f4e8cc6450" } new_table_name: "default.Table_19e8169b84d0438ba2b732151d85cf52" modify_external_catalogs: true
I20250905 19:52:41.057441 2340 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_ae712f1c45694ed9b3b777066df1d465" } new_table_name: "default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431" modify_external_catalogs: true
I20250905 19:52:41.065137 907 raft_consensus.cc:2466] T 5a5d38136c034b8497045176f749a98d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.067399 2397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_5462efe018cb409ea03a08a7b129264a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.068032 2397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5462efe018cb409ea03a08a7b129264a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.069821 2405 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_19682dbce0bc4480b82d39235ba7f55b" } modify_external_catalogs: true
I20250905 19:52:41.070103 2405 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_19682dbce0bc4480b82d39235ba7f55b" } modify_external_catalogs: true
I20250905 19:52:41.070736 3297 raft_consensus.cc:491] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.070905 3297 raft_consensus.cc:513] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.071480 3297 leader_election.cc:290] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.076174 3346 raft_consensus.cc:491] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.076318 3346 raft_consensus.cc:513] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.076920 3346 leader_election.cc:290] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:41.077364 3346 raft_consensus.cc:491] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.077471 3346 raft_consensus.cc:513] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.078011 3346 leader_election.cc:290] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:41.078627 3346 raft_consensus.cc:491] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.078749 3346 raft_consensus.cc:513] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.079320 3346 leader_election.cc:290] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:41.080024 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ad956757086b4d728698679c63c9b7ee" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.080219 767 raft_consensus.cc:3058] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.069114 960 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:41.083415 767 raft_consensus.cc:2466] T ad956757086b4d728698679c63c9b7ee P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.084100 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.094620 770 raft_consensus.cc:3058] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.095136 764 raft_consensus.cc:3058] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.101359 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.104036 2373 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet dcf448f9abc348df8d2d2501420d8f45
I20250905 19:52:41.118745 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.120133 984 leader_election.cc:304] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
W20250905 19:52:41.120575 984 leader_election.cc:343] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.121078 3299 raft_consensus.cc:2802] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.121243 3299 raft_consensus.cc:695] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.121660 2496 consensus_queue.cc:1035] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.135710 3299 consensus_queue.cc:237] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:41.136646 2372 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet c4e641ac5b5c49fda12450bbf1fec2b4 has no consensus state
I20250905 19:52:41.144309 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_3f0abddc641a42f590165ca4f321a535"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.144968 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3f0abddc641a42f590165ca4f321a535 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.151787 3346 raft_consensus.cc:491] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.152007 3346 raft_consensus.cc:513] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.152781 3346 leader_election.cc:290] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:41.164933 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 43927.8us
I20250905 19:52:41.165252 2392 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_21f0cf32be59451d987d978191525180" } modify_external_catalogs: true
W20250905 19:52:41.165428 984 leader_election.cc:343] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.165383 2496 consensus_queue.cc:1035] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.165452 2392 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_21f0cf32be59451d987d978191525180" } modify_external_catalogs: true
W20250905 19:52:41.165710 984 leader_election.cc:343] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:41.169449 2285 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet c4e641ac5b5c49fda12450bbf1fec2b4 has no consensus state
I20250905 19:52:41.170250 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:41.170866 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:41.171378 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edb9379335ca4eb5a32a6cbe85601a1f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:41.171583 1048 raft_consensus.cc:2466] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:41.174037 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 with delete_type TABLET_DATA_DELETED (replaced by tablet 350197e134c549198771f4f8337d5dfc) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.177625 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
/tmp/dist-test-taskzbeQlP/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:34395
--webserver_interface=127.0.104.60
--webserver_port=44671
--builtin_ntp_servers=127.0.104.20:42539
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--hive_metastore_notification_log_poll_period_seconds=1
--leader_failure_max_missed_heartbeat_periods=1.0
--log_preallocate_segments=false with env {}
I20250905 19:52:41.184111 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 with delete_type TABLET_DATA_DELETED (replaced by tablet 26905425077e4746bccb51431da706c9) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.204830 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d with delete_type TABLET_DATA_DELETED (replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.207504 739 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf448f9abc348df8d2d2501420d8f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
W20250905 19:52:41.207465 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 155101us
I20250905 19:52:41.207726 986 leader_election.cc:304] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:41.208012 3274 raft_consensus.cc:2802] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.208083 3274 raft_consensus.cc:491] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.208154 3274 raft_consensus.cc:3058] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.211248 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c8e53152ada84954b12bbfe9d1fe3c16" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.211505 907 raft_consensus.cc:2391] T c8e53152ada84954b12bbfe9d1fe3c16 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:41.212538 3274 raft_consensus.cc:513] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.213151 3274 leader_election.cc:290] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.213841 986 leader_election.cc:304] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:41.214064 3274 raft_consensus.cc:2802] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.214125 3274 raft_consensus.cc:491] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.214195 3274 raft_consensus.cc:3058] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.216822 3274 raft_consensus.cc:513] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.218365 3274 leader_election.cc:290] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.219839 986 leader_election.cc:304] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:41.220113 3297 raft_consensus.cc:2802] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.220225 3297 raft_consensus.cc:491] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.220350 3297 raft_consensus.cc:3058] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.222808 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6ee9f4fe3a464abaaca48f22ea183c0e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.223026 904 raft_consensus.cc:2391] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:41.223534 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.223682 907 raft_consensus.cc:2466] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.225646 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.225812 907 raft_consensus.cc:2391] T 31484264112d42b089536d1ecc8ba021 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:41.228514 3539 tablet_replica.cc:331] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:41.228998 3539 raft_consensus.cc:2241] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.229264 3539 raft_consensus.cc:2270] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.229231 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10fef848b9b746c5a0de6a8c3ca74f7a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.229372 907 raft_consensus.cc:3058] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.230002 3539 ts_tablet_manager.cc:1905] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.232002 907 raft_consensus.cc:2466] T 10fef848b9b746c5a0de6a8c3ca74f7a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.232661 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.232795 907 raft_consensus.cc:3058] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:41.235584 712 leader_election.cc:343] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:41.235939 907 raft_consensus.cc:2466] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.236621 3346 consensus_queue.cc:1035] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.237953 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 with delete_type TABLET_DATA_DELETED (replaced by tablet 350197e134c549198771f4f8337d5dfc) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.240307 3274 consensus_queue.cc:1035] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.242074 986 leader_election.cc:304] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:41.243252 3299 raft_consensus.cc:2802] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.243357 3299 raft_consensus.cc:491] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.243434 3299 raft_consensus.cc:3058] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.247721 3299 raft_consensus.cc:513] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.247820 2397 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_4efd1c77ccaf4d3bb149d66fd3e4da5f" } new_table_name: "default.Table_a0475758d8b449db8b0d9338b8d2f0f1" modify_external_catalogs: true
I20250905 19:52:41.248339 3299 leader_election.cc:290] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.249382 3534 consensus_queue.cc:1035] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.250654 1170 tablet_bootstrap.cc:492] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:41.250591 3536 consensus_queue.cc:1035] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.252204 3297 raft_consensus.cc:513] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.260891 2373 catalog_manager.cc:5582] T a551e641a19f4d3f924fb176c4a25ca7 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.261315 2373 catalog_manager.cc:5582] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.261596 2373 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:41.261889 2373 catalog_manager.cc:5582] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.262311 2373 catalog_manager.cc:5582] T 7e8bfe047a6e4cc087eea75244da5b41 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.257164 3346 consensus_queue.cc:1035] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.264804 1170 ts_tablet_manager.cc:1397] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.274s user 0.043s sys 0.067s
I20250905 19:52:41.276175 739 tablet_service.cc:1468] Processing CreateTablet for tablet 3c5cf2bc19b043f68838f8a6af6393cd (DEFAULT_TABLE table=default.Table_c6be55785e1244cb85764306f76c04a2 [id=5ee8596f32f042698dfbe530710fda20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.256206 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "31484264112d42b089536d1ecc8ba021" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:41.276607 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5cf2bc19b043f68838f8a6af6393cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.259115 1099 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:41.276134 751 tablet_service.cc:1468] Processing CreateTablet for tablet 7a104b1c8ce14fa2bf101e1b34183cfa (DEFAULT_TABLE table=default.Table_6e40598f9dbc40ac9860eb5b78130a93 [id=408552212a1f42089f1d4cb87b4a13b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.282996 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a104b1c8ce14fa2bf101e1b34183cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.283372 986 leader_election.cc:304] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.284268 3536 raft_consensus.cc:2802] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.284399 3536 raft_consensus.cc:695] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.284626 3536 consensus_queue.cc:237] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.286757 3297 leader_election.cc:290] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.287585 739 tablet_service.cc:1468] Processing CreateTablet for tablet 98db7e2a9d6d4f4d984388a493295d72 (DEFAULT_TABLE table=default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.288022 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98db7e2a9d6d4f4d984388a493295d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.289597 3542 consensus_queue.cc:1035] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.290452 711 leader_election.cc:304] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:41.291225 711 leader_election.cc:304] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:41.291497 3346 raft_consensus.cc:3058] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.293008 1170 raft_consensus.cc:357] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.293166 1170 raft_consensus.cc:383] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.293213 1170 raft_consensus.cc:738] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.293503 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 with delete_type TABLET_DATA_DELETED (replaced by tablet 26905425077e4746bccb51431da706c9) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.293979 3346 raft_consensus.cc:2747] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:41.294037 711 leader_election.cc:304] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:41.294631 3346 raft_consensus.cc:2802] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.294705 3346 raft_consensus.cc:491] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.294775 3346 raft_consensus.cc:3058] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.294986 2372 master_service.cc:948] ReplaceTablet: received request to replace tablet d1a11777cf2d4ea9a5cdf792f73aff76 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:41.295397 711 leader_election.cc:304] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:41.296424 3299 raft_consensus.cc:491] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.296537 3299 raft_consensus.cc:513] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.296819 3282 raft_consensus.cc:3058] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.297073 3299 leader_election.cc:290] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.297420 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d with delete_type TABLET_DATA_DELETED (replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.298682 878 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf448f9abc348df8d2d2501420d8f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.299108 1170 consensus_queue.cc:260] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.299405 3282 raft_consensus.cc:2747] T 31484264112d42b089536d1ecc8ba021 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:41.300016 878 tablet_service.cc:1468] Processing CreateTablet for tablet 7a104b1c8ce14fa2bf101e1b34183cfa (DEFAULT_TABLE table=default.Table_6e40598f9dbc40ac9860eb5b78130a93 [id=408552212a1f42089f1d4cb87b4a13b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.300619 744 tablet_service.cc:1468] Processing CreateTablet for tablet 1110897f9ec14741b9b882b6e09081ab (DEFAULT_TABLE table=default.Table_b24d9f16949145cda6b4f82d78993fa0 [id=2242efd05b0e4fae989ce99e9431aa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.300982 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1110897f9ec14741b9b882b6e09081ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.301066 891 tablet_service.cc:1468] Processing CreateTablet for tablet 3c5cf2bc19b043f68838f8a6af6393cd (DEFAULT_TABLE table=default.Table_c6be55785e1244cb85764306f76c04a2 [id=5ee8596f32f042698dfbe530710fda20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.301707 3346 raft_consensus.cc:513] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.304240 893 tablet_service.cc:1468] Processing CreateTablet for tablet 98db7e2a9d6d4f4d984388a493295d72 (DEFAULT_TABLE table=default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.304652 3346 leader_election.cc:290] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:41.305306 882 tablet_service.cc:1468] Processing CreateTablet for tablet 1110897f9ec14741b9b882b6e09081ab (DEFAULT_TABLE table=default.Table_b24d9f16949145cda6b4f82d78993fa0 [id=2242efd05b0e4fae989ce99e9431aa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.306288 1019 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 with delete_type TABLET_DATA_DELETED (replaced by tablet 350197e134c549198771f4f8337d5dfc) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.306447 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d with delete_type TABLET_DATA_DELETED (replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.306582 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 with delete_type TABLET_DATA_DELETED (replaced by tablet 26905425077e4746bccb51431da706c9) from {username='slave'} at 127.0.0.1:50082
W20250905 19:52:41.307101 848 consensus_peers.cc:489] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:41.307144 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf448f9abc348df8d2d2501420d8f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.307435 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 7a104b1c8ce14fa2bf101e1b34183cfa (DEFAULT_TABLE table=default.Table_6e40598f9dbc40ac9860eb5b78130a93 [id=408552212a1f42089f1d4cb87b4a13b5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.307806 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a104b1c8ce14fa2bf101e1b34183cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.314754 875 tablet_service.cc:1468] Processing CreateTablet for tablet 7d2e58cc18b94f9ab036e3376920351c (DEFAULT_TABLE table=default.Table_d090d4e81b7c45e49c707734cfeae659 [id=02eadd9157e44aaf861adbfd95b2fc9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.314775 881 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c (DEFAULT_TABLE table=default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.315433 883 tablet_service.cc:1468] Processing CreateTablet for tablet e476304b492f4bd69d5ea0f109263517 (DEFAULT_TABLE table=default.Table_3e897c08965d4265b2a5937b15f5c715 [id=48d7bc33c38d44fc977294a75d895a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.316092 889 tablet_service.cc:1468] Processing CreateTablet for tablet fe00969eb9654976a0efeac2fa7c91d3 (DEFAULT_TABLE table=default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.316164 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7d2e58cc18b94f9ab036e3376920351c (DEFAULT_TABLE table=default.Table_d090d4e81b7c45e49c707734cfeae659 [id=02eadd9157e44aaf861adbfd95b2fc9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.316524 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2e58cc18b94f9ab036e3376920351c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.316911 877 tablet_service.cc:1468] Processing CreateTablet for tablet 736bfce682e64498a8dde3310305536a (DEFAULT_TABLE table=default.Table_6a2d2733af51406295e2d14f82f6cb1e [id=edb5d0cff27f4181a256a126d5319d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.317019 892 tablet_service.cc:1468] Processing CreateTablet for tablet 9ca3fc7055c44df282285806ea151e51 (DEFAULT_TABLE table=default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.317781 894 tablet_service.cc:1468] Processing CreateTablet for tablet 4a95130c82e14bbcae577d4244b876ab (DEFAULT_TABLE table=default.Table_00d9a0127df4444eb081316b26c6ec87 [id=820565dc3d0942839324e01101b02e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.320679 986 leader_election.cc:304] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.321550 743 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c (DEFAULT_TABLE table=default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.321905 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3e1732bff14d49ade14fbfa6a9784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.324381 3534 raft_consensus.cc:2802] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.324512 3534 raft_consensus.cc:695] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.324891 3534 consensus_queue.cc:237] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.321460 1169 tablet_bootstrap.cc:492] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:41.327610 1169 ts_tablet_manager.cc:1397] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.291s user 0.015s sys 0.031s
I20250905 19:52:41.328372 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98db7e2a9d6d4f4d984388a493295d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.328454 1169 raft_consensus.cc:357] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.328613 1169 raft_consensus.cc:383] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.328670 1169 raft_consensus.cc:738] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.328840 1169 consensus_queue.cc:260] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.329320 743 tablet_service.cc:1468] Processing CreateTablet for tablet e476304b492f4bd69d5ea0f109263517 (DEFAULT_TABLE table=default.Table_3e897c08965d4265b2a5937b15f5c715 [id=48d7bc33c38d44fc977294a75d895a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.329684 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e476304b492f4bd69d5ea0f109263517. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.329761 1169 ts_tablet_manager.cc:1428] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:41.330011 1169 tablet_bootstrap.cc:492] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:41.331889 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a95130c82e14bbcae577d4244b876ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.332238 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe00969eb9654976a0efeac2fa7c91d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.338974 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.339143 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca3fc7055c44df282285806ea151e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.339174 911 raft_consensus.cc:3058] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.343029 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.343228 905 raft_consensus.cc:3058] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.343703 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a104b1c8ce14fa2bf101e1b34183cfa. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.344197 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5cf2bc19b043f68838f8a6af6393cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.344431 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736bfce682e64498a8dde3310305536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.344636 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3e1732bff14d49ade14fbfa6a9784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.346374 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e476304b492f4bd69d5ea0f109263517. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.348233 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2e58cc18b94f9ab036e3376920351c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.356691 753 tablet_service.cc:1468] Processing CreateTablet for tablet fe00969eb9654976a0efeac2fa7c91d3 (DEFAULT_TABLE table=default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.357090 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe00969eb9654976a0efeac2fa7c91d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.358539 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1110897f9ec14741b9b882b6e09081ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.361846 904 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.362077 904 raft_consensus.cc:2466] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.362597 1170 ts_tablet_manager.cc:1428] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.087s user 0.001s sys 0.001s
I20250905 19:52:41.362929 1170 tablet_bootstrap.cc:492] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:41.364930 1170 tablet_bootstrap.cc:654] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.366178 986 leader_election.cc:304] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.366526 3536 raft_consensus.cc:2802] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.366633 3536 raft_consensus.cc:491] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.366729 3536 raft_consensus.cc:3058] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.370139 753 tablet_service.cc:1468] Processing CreateTablet for tablet 736bfce682e64498a8dde3310305536a (DEFAULT_TABLE table=default.Table_6a2d2733af51406295e2d14f82f6cb1e [id=edb5d0cff27f4181a256a126d5319d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.371461 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736bfce682e64498a8dde3310305536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.374817 905 raft_consensus.cc:2466] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.380849 911 raft_consensus.cc:2466] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.384467 742 tablet_service.cc:1468] Processing CreateTablet for tablet 9ca3fc7055c44df282285806ea151e51 (DEFAULT_TABLE table=default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.384611 3151 consensus_queue.cc:1035] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.384980 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca3fc7055c44df282285806ea151e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.385349 986 leader_election.cc:304] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.388659 3297 raft_consensus.cc:2802] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.388787 3297 raft_consensus.cc:695] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.389019 3297 consensus_queue.cc:237] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.389626 986 leader_election.cc:304] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.389920 3151 raft_consensus.cc:2802] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.390103 3151 raft_consensus.cc:695] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.390416 3151 consensus_queue.cc:237] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.394364 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 3c5cf2bc19b043f68838f8a6af6393cd (DEFAULT_TABLE table=default.Table_c6be55785e1244cb85764306f76c04a2 [id=5ee8596f32f042698dfbe530710fda20]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.394757 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3c5cf2bc19b043f68838f8a6af6393cd. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.396332 742 tablet_service.cc:1468] Processing CreateTablet for tablet 4a95130c82e14bbcae577d4244b876ab (DEFAULT_TABLE table=default.Table_00d9a0127df4444eb081316b26c6ec87 [id=820565dc3d0942839324e01101b02e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.396831 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a95130c82e14bbcae577d4244b876ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.400637 3536 raft_consensus.cc:513] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.409190 3536 leader_election.cc:290] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.413465 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "10fef848b9b746c5a0de6a8c3ca74f7a" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.414453 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.418273 3547 tablet_replica.cc:331] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.419781 3547 raft_consensus.cc:2241] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.420236 3547 raft_consensus.cc:2270] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.420796 3542 raft_consensus.cc:2747] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:41.421008 3547 ts_tablet_manager.cc:1905] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.422479 3541 tablet_replica.cc:331] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:41.422816 3541 raft_consensus.cc:2241] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:41.424387 3541 raft_consensus.cc:2270] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.424837 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 1110897f9ec14741b9b882b6e09081ab (DEFAULT_TABLE table=default.Table_b24d9f16949145cda6b4f82d78993fa0 [id=2242efd05b0e4fae989ce99e9431aa4b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.425179 3541 ts_tablet_manager.cc:1905] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.425354 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1110897f9ec14741b9b882b6e09081ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.432020 1030 tablet_service.cc:1468] Processing CreateTablet for tablet 7d2e58cc18b94f9ab036e3376920351c (DEFAULT_TABLE table=default.Table_d090d4e81b7c45e49c707734cfeae659 [id=02eadd9157e44aaf861adbfd95b2fc9c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.432416 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d2e58cc18b94f9ab036e3376920351c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.432963 1169 tablet_bootstrap.cc:654] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.432008 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 98db7e2a9d6d4f4d984388a493295d72 (DEFAULT_TABLE table=default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.435657 1026 tablet_service.cc:1468] Processing CreateTablet for tablet e476304b492f4bd69d5ea0f109263517 (DEFAULT_TABLE table=default.Table_3e897c08965d4265b2a5937b15f5c715 [id=48d7bc33c38d44fc977294a75d895a15]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.435920 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98db7e2a9d6d4f4d984388a493295d72. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.436158 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e476304b492f4bd69d5ea0f109263517. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.436225 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.436394 909 raft_consensus.cc:3058] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.436964 2496 raft_consensus.cc:491] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.437074 2496 raft_consensus.cc:513] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.437566 2496 leader_election.cc:290] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.438957 909 raft_consensus.cc:2466] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.439908 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c (DEFAULT_TABLE table=default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.440099 986 leader_election.cc:304] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.440163 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.440269 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6e3e1732bff14d49ade14fbfa6a9784c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.440335 2931 raft_consensus.cc:2802] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.440387 905 raft_consensus.cc:2466] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.440443 2931 raft_consensus.cc:695] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.440677 2931 consensus_queue.cc:237] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.442655 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.442881 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.452232 1029 tablet_service.cc:1468] Processing CreateTablet for tablet fe00969eb9654976a0efeac2fa7c91d3 (DEFAULT_TABLE table=default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.452642 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe00969eb9654976a0efeac2fa7c91d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.460505 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 9ca3fc7055c44df282285806ea151e51 (DEFAULT_TABLE table=default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.461035 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9ca3fc7055c44df282285806ea151e51. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.463995 2405 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d709e2b4f10149a1818c1e1431b95937
I20250905 19:52:41.464668 2405 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_191f4f9b06184a8a8429cce5802c520e" } modify_external_catalogs: true
I20250905 19:52:41.464809 2405 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_191f4f9b06184a8a8429cce5802c520e" } modify_external_catalogs: true
I20250905 19:52:41.465162 2405 master_service.cc:948] ReplaceTablet: received request to replace tablet be1315824fd749049a62af3e6a113b9a from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:41.467756 3557 tablet_bootstrap.cc:492] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:41.468170 2372 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d1a11777cf2d4ea9a5cdf792f73aff76
I20250905 19:52:41.468293 3547 ts_tablet_manager.cc:1918] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:41.468401 3547 log.cc:1199] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/5fdc9d9815864aa7bf5ca9a5c4485390
I20250905 19:52:41.468650 2372 catalog_manager.cc:6652] ReplaceTablet: tablet d1a11777cf2d4ea9a5cdf792f73aff76 deleted and replaced by tablet bd3a516f3a364c9ba30fb58b1baed332
I20250905 19:52:41.468789 3547 ts_tablet_manager.cc:1939] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:41.468966 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_f1b2c338b96549e0b2fa23f258256b57"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.469524 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f1b2c338b96549e0b2fa23f258256b57 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.469758 3547 tablet_replica.cc:331] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.469945 3547 raft_consensus.cc:2241] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.470132 3547 raft_consensus.cc:2270] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.470634 3547 ts_tablet_manager.cc:1905] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.475229 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 4a95130c82e14bbcae577d4244b876ab (DEFAULT_TABLE table=default.Table_00d9a0127df4444eb081316b26c6ec87 [id=820565dc3d0942839324e01101b02e11]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.476950 822 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:41.475590 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4a95130c82e14bbcae577d4244b876ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.485059 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_6ebce8ef802e4d9682e7815b99d7e8db"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.485692 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6ebce8ef802e4d9682e7815b99d7e8db in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.490209 753 tablet_service.cc:1515] Processing DeleteTablet for tablet d709e2b4f10149a1818c1e1431b95937 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.490659 753 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 with delete_type TABLET_DATA_DELETED (replaced by tablet bd3a516f3a364c9ba30fb58b1baed332) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.491173 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 736bfce682e64498a8dde3310305536a (DEFAULT_TABLE table=default.Table_6a2d2733af51406295e2d14f82f6cb1e [id=edb5d0cff27f4181a256a126d5319d55]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.491622 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 736bfce682e64498a8dde3310305536a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.493321 3547 ts_tablet_manager.cc:1918] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:41.494073 3547 log.cc:1199] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/89a73e984cbb4268b563f1ca01f4c65d
I20250905 19:52:41.494493 3547 ts_tablet_manager.cc:1939] T 89a73e984cbb4268b563f1ca01f4c65d P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:41.496872 3547 tablet_replica.cc:331] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.497136 3547 raft_consensus.cc:2241] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.497345 3547 raft_consensus.cc:2270] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.501608 3547 ts_tablet_manager.cc:1905] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.505424 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "edb9379335ca4eb5a32a6cbe85601a1f" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:41.505695 1052 raft_consensus.cc:3058] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.506115 2405 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet be1315824fd749049a62af3e6a113b9a
I20250905 19:52:41.506573 2405 catalog_manager.cc:6652] ReplaceTablet: tablet be1315824fd749049a62af3e6a113b9a deleted and replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf
I20250905 19:52:41.508611 3547 ts_tablet_manager.cc:1918] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:41.508737 3547 log.cc:1199] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9a81376d4c0b47a9817deae4d34ffb47
I20250905 19:52:41.509074 3547 ts_tablet_manager.cc:1939] T 9a81376d4c0b47a9817deae4d34ffb47 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:41.509963 3547 tablet_replica.cc:331] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.510182 3547 raft_consensus.cc:2241] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.510350 3547 raft_consensus.cc:2270] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.510888 3547 ts_tablet_manager.cc:1905] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.511207 1052 raft_consensus.cc:2466] T edb9379335ca4eb5a32a6cbe85601a1f P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:41.515693 2403 catalog_manager.cc:5582] T 5302b79208494e48a5f2778c5f52ad01 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.518318 753 tablet_service.cc:1515] Processing DeleteTablet for tablet be1315824fd749049a62af3e6a113b9a with delete_type TABLET_DATA_DELETED (replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.518913 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:41.519320 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:41.519955 2403 catalog_manager.cc:5582] T 180437d1534946619384307ddcb7c937 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.521930 2403 catalog_manager.cc:5582] T d59da39e7bcf497a93fb9174328c1814 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.522421 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:41.523362 1052 raft_consensus.cc:1273] T f34f0b29d6104cac8ab35309fd1edae8 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:52:41.524082 2403 catalog_manager.cc:5582] T e5fc2c2ba49444af939f5a975165362b P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.532830 3547 ts_tablet_manager.cc:1918] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:41.533059 3547 log.cc:1199] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/dcf448f9abc348df8d2d2501420d8f45
I20250905 19:52:41.533479 3547 ts_tablet_manager.cc:1939] T dcf448f9abc348df8d2d2501420d8f45 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:41.537796 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 97723.9us
I20250905 19:52:41.538123 986 leader_election.cc:304] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.538349 3274 raft_consensus.cc:2802] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.538442 3274 raft_consensus.cc:491] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.538507 3274 raft_consensus.cc:3058] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.542249 3274 raft_consensus.cc:513] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.542819 3274 leader_election.cc:290] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.547101 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.547461 911 raft_consensus.cc:3058] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.547528 1169 tablet_bootstrap.cc:492] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:41.547662 1169 ts_tablet_manager.cc:1397] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.218s user 0.002s sys 0.000s
I20250905 19:52:41.548894 1170 tablet_bootstrap.cc:492] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:41.549041 1170 ts_tablet_manager.cc:1397] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.186s user 0.003s sys 0.000s
I20250905 19:52:41.549767 1170 raft_consensus.cc:357] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.549944 1170 raft_consensus.cc:383] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.549999 1170 raft_consensus.cc:738] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.550208 1170 consensus_queue.cc:260] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.550693 1170 ts_tablet_manager.cc:1428] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:41.550951 1170 tablet_bootstrap.cc:492] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:41.553686 3539 ts_tablet_manager.cc:1918] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:41.553787 3539 log.cc:1199] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/5fdc9d9815864aa7bf5ca9a5c4485390
I20250905 19:52:41.554188 3539 ts_tablet_manager.cc:1939] T 5fdc9d9815864aa7bf5ca9a5c4485390 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:41.555263 3539 tablet_replica.cc:331] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:41.555469 3539 raft_consensus.cc:2241] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.556314 3299 consensus_queue.cc:1035] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.557682 3536 consensus_queue.cc:1035] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.560573 1169 raft_consensus.cc:357] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.560736 1169 raft_consensus.cc:383] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.560788 1169 raft_consensus.cc:738] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.560971 1169 consensus_queue.cc:260] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.556648 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]) successfully deleted
I20250905 19:52:41.561616 1169 ts_tablet_manager.cc:1428] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.014s user 0.002s sys 0.000s
I20250905 19:52:41.561898 1169 tablet_bootstrap.cc:492] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:41.565392 984 leader_election.cc:343] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:41.565711 984 leader_election.cc:343] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.567768 3557 tablet_bootstrap.cc:654] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.554052 2403 catalog_manager.cc:5582] T 160a7814abf9423dbe965c252c20fa7f P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.575227 2403 catalog_manager.cc:5582] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.576301 2403 catalog_manager.cc:5582] T 65368306f86845008d4d486023e1fc29 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.579036 2372 master_service.cc:948] ReplaceTablet: received request to replace tablet 5a59d7fb5dc54b479d965282b1c7d24b from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:41.579188 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_b0e8d7317a7e41f7a3e73f23fe162bbe"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.579833 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_b0e8d7317a7e41f7a3e73f23fe162bbe in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.580610 2403 catalog_manager.cc:5582] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.581122 2403 catalog_manager.cc:5582] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.581451 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
W20250905 19:52:41.583346 712 leader_election.cc:343] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:41.583554 3542 consensus_queue.cc:1035] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.586652 911 raft_consensus.cc:2466] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:41.589233 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_3d26023ae0754b29bb4dfede839e5998"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.590903 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3d26023ae0754b29bb4dfede839e5998 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.581861 2403 catalog_manager.cc:5582] T 5b4f9e12781a46f19b563746a70b59e4 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.593168 2403 catalog_manager.cc:5582] T 32fa16972f2a42d39b338b833c185b28 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.593588 2403 catalog_manager.cc:5582] T f6f29ac5e68d4d8da8f3496fefe695ec P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.594077 2403 catalog_manager.cc:5582] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.596163 3297 raft_consensus.cc:491] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.596311 3297 raft_consensus.cc:513] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.596913 3297 leader_election.cc:290] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.598026 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.598217 908 raft_consensus.cc:2466] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.598724 986 leader_election.cc:304] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.599329 3297 raft_consensus.cc:2802] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.599443 3297 raft_consensus.cc:695] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.599682 3297 consensus_queue.cc:237] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.600291 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_5cd8ba66cf734db5afca42d7fbf46281"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:41.594415 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:41.601291 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
W20250905 19:52:41.601260 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5cd8ba66cf734db5afca42d7fbf46281 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.601639 2403 catalog_manager.cc:5582] T 91396422d0bb468395db04aa80262882 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.604501 2403 catalog_manager.cc:5582] T b1f46261822442e0a961d2d63bdb35c8 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.605314 986 leader_election.cc:304] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.605475 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:41.605872 2403 catalog_manager.cc:5582] T ca646f7c162e41469f283657e72823b4 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.606079 3536 raft_consensus.cc:2802] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.606155 3536 raft_consensus.cc:491] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.606218 3536 raft_consensus.cc:3058] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.606448 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:41.607368 2403 catalog_manager.cc:5582] T 923ad8e40a07409e8f6a0152a0b2ef75 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.608292 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:41.608947 3536 raft_consensus.cc:513] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.609333 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:41.609501 3536 leader_election.cc:290] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:41.612007 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 46073.6us
I20250905 19:52:41.610222 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_fe6d663d69f74f929b1255e176be4ead"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.616551 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe6d663d69f74f929b1255e176be4ead in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.619493 1169 tablet_bootstrap.cc:654] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:41.620875 984 leader_election.cc:343] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:41.621212 984 leader_election.cc:343] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.622226 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.623286 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.624727 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.627125 711 leader_election.cc:304] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:41.627501 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]) successfully deleted
I20250905 19:52:41.627872 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]) successfully deleted
I20250905 19:52:41.628028 3542 raft_consensus.cc:2802] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.628129 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]) successfully deleted
I20250905 19:52:41.628288 3546 consensus_queue.cc:1035] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
I20250905 19:52:41.628429 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet dcf448f9abc348df8d2d2501420d8f45 (table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]) successfully deleted
I20250905 19:52:41.628793 3297 raft_consensus.cc:491] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.628930 3297 raft_consensus.cc:513] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.629501 3297 leader_election.cc:290] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.629801 3297 raft_consensus.cc:491] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.629925 3297 raft_consensus.cc:513] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.630363 3297 leader_election.cc:290] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.632251 2403 catalog_manager.cc:5582] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.632570 3297 raft_consensus.cc:491] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.632658 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:41.632719 3297 raft_consensus.cc:513] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.633314 3297 leader_election.cc:290] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.638898 3541 ts_tablet_manager.cc:1918] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:41.639093 3541 log.cc:1199] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/5fdc9d9815864aa7bf5ca9a5c4485390
I20250905 19:52:41.639578 3541 ts_tablet_manager.cc:1939] T 5fdc9d9815864aa7bf5ca9a5c4485390 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:41.644696 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.645073 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.645183 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:41.652684 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:41.653537 2403 catalog_manager.cc:5582] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.654013 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:41.654790 2403 catalog_manager.cc:5582] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.660065 3539 raft_consensus.cc:2270] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.660979 3539 ts_tablet_manager.cc:1905] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.668031 2403 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:41.668474 2403 catalog_manager.cc:5582] T fa5d34f9c0ee4d55ade61b30b0f4c293 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.669353 2403 catalog_manager.cc:5582] T 862214a31059407baefef8a390aaea69 P 835daf30e4324782848290be181ae78f reported cstate change: term changed from 0 to 1, leader changed from <none> to 835daf30e4324782848290be181ae78f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "835daf30e4324782848290be181ae78f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.663000 1170 tablet_bootstrap.cc:654] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.676474 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d709e2b4f10149a1818c1e1431b95937 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.678216 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 with delete_type TABLET_DATA_DELETED (replaced by tablet bd3a516f3a364c9ba30fb58b1baed332) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.678610 3547 tablet_replica.cc:331] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.678879 3547 raft_consensus.cc:2241] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:41.679282 3547 raft_consensus.cc:2270] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.680121 3547 ts_tablet_manager.cc:1905] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.689216 3547 ts_tablet_manager.cc:1918] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:41.689450 3547 log.cc:1199] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d1a11777cf2d4ea9a5cdf792f73aff76
I20250905 19:52:41.689914 3547 ts_tablet_manager.cc:1939] T d1a11777cf2d4ea9a5cdf792f73aff76 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:41.700242 2372 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 5a59d7fb5dc54b479d965282b1c7d24b
I20250905 19:52:41.700739 2372 catalog_manager.cc:6652] ReplaceTablet: tablet 5a59d7fb5dc54b479d965282b1c7d24b deleted and replaced by tablet b23111dc8af94b56af9538f973248f2a
I20250905 19:52:41.701148 2372 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_bc301cb31d184494ad445294416d0a20" } modify_external_catalogs: true
I20250905 19:52:41.701354 2372 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_bc301cb31d184494ad445294416d0a20" } modify_external_catalogs: true
I20250905 19:52:41.709373 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b with delete_type TABLET_DATA_DELETED (replaced by tablet b23111dc8af94b56af9538f973248f2a) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:41.640192 894 tablet_service.cc:1515] Processing DeleteTablet for tablet d709e2b4f10149a1818c1e1431b95937 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.640239 877 tablet_service.cc:1515] Processing DeleteTablet for tablet be1315824fd749049a62af3e6a113b9a with delete_type TABLET_DATA_DELETED (replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.640339 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 with delete_type TABLET_DATA_DELETED (replaced by tablet bd3a516f3a364c9ba30fb58b1baed332) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.711198 3541 tablet_replica.cc:331] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:41.711658 1170 tablet_bootstrap.cc:492] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:41.711910 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet be1315824fd749049a62af3e6a113b9a with delete_type TABLET_DATA_DELETED (replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.712284 3547 tablet_replica.cc:331] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.712585 3547 raft_consensus.cc:2241] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:41.712949 3547 raft_consensus.cc:2270] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.713644 3547 ts_tablet_manager.cc:1905] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.716032 1170 ts_tablet_manager.cc:1397] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.165s user 0.004s sys 0.000s
W20250905 19:52:41.717483 2291 catalog_manager.cc:4895] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet d709e2b4f10149a1818c1e1431b95937 because the tablet was not found. No further retry: Not found: Tablet not found: d709e2b4f10149a1818c1e1431b95937
I20250905 19:52:41.717154 767 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9ae8fd5fdc4b539308268ba9729cd6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.717550 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d1d24868204ce09a7f5e40357fec18" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.717885 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d1a11777cf2d4ea9a5cdf792f73aff76 (table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]) successfully deleted
I20250905 19:52:41.718688 1170 raft_consensus.cc:357] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.718832 1170 raft_consensus.cc:383] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.718886 1170 raft_consensus.cc:738] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.716399 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b with delete_type TABLET_DATA_DELETED (replaced by tablet b23111dc8af94b56af9538f973248f2a) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:41.719247 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.719736 764 raft_consensus.cc:3058] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.704456 3557 tablet_bootstrap.cc:492] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:41.723076 3557 ts_tablet_manager.cc:1397] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.255s user 0.003s sys 0.001s
I20250905 19:52:41.723938 3557 raft_consensus.cc:357] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.724220 3557 raft_consensus.cc:383] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.724323 3557 raft_consensus.cc:738] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.724571 3557 consensus_queue.cc:260] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.725283 3557 ts_tablet_manager.cc:1428] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:41.725652 3557 tablet_bootstrap.cc:492] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:41.726620 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 58490.5us
I20250905 19:52:41.726979 3541 raft_consensus.cc:2241] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:41.730599 3541 raft_consensus.cc:2270] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.732023 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 5fdc9d9815864aa7bf5ca9a5c4485390 (table default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]) successfully deleted
W20250905 19:52:41.733839 984 leader_election.cc:343] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:41.734228 984 leader_election.cc:343] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.732349 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b with delete_type TABLET_DATA_DELETED (replaced by tablet b23111dc8af94b56af9538f973248f2a) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:41.732873 911 raft_consensus.cc:1273] T f34f0b29d6104cac8ab35309fd1edae8 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:52:41.735183 3541 ts_tablet_manager.cc:1905] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.736860 1170 consensus_queue.cc:260] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.737545 3557 tablet_bootstrap.cc:654] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.739246 1170 ts_tablet_manager.cc:1428] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.021s user 0.001s sys 0.000s
I20250905 19:52:41.739871 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:41.739989 1170 tablet_bootstrap.cc:492] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:41.740077 908 raft_consensus.cc:3058] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.740623 3546 consensus_queue.cc:1035] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.742589 908 raft_consensus.cc:2466] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:41.742652 711 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 114882us
I20250905 19:52:41.739349 3542 raft_consensus.cc:695] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:41.743639 3542 consensus_queue.cc:237] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.748507 3557 tablet_bootstrap.cc:492] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:41.751904 3346 raft_consensus.cc:491] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.752130 3346 raft_consensus.cc:513] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:41.754004 712 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: fd1280fefc3e43db9928567a734644f1
I20250905 19:52:41.754930 3346 leader_election.cc:290] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:41.748525 893 tablet_service.cc:1468] Processing CreateTablet for tablet c200681d4d77425389558e6489fb6032 (DEFAULT_TABLE table=default.Table_870269fff0a1492e9a170850a3981347 [id=d4438b648a964ba6841ff82623ee18fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.749024 877 tablet_service.cc:1468] Processing CreateTablet for tablet 2edf591963ca46369c29a3fb2bf24bc3 (DEFAULT_TABLE table=default.Table_a0b7f5e829264dc5974e4d1039940a52 [id=c683387027d54cca98217b7398feed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.749349 894 tablet_service.cc:1468] Processing CreateTablet for tablet cea5667d882b4b57a6fa4e9866641bf6 (DEFAULT_TABLE table=default.Table_ce82d3ecb4894b59aa7e82b457011d4a [id=68581ab2921a42d1931a9edb44fa1ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.756450 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c200681d4d77425389558e6489fb6032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.756685 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea5667d882b4b57a6fa4e9866641bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.760102 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2edf591963ca46369c29a3fb2bf24bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.760102 753 tablet_service.cc:1468] Processing CreateTablet for tablet c200681d4d77425389558e6489fb6032 (DEFAULT_TABLE table=default.Table_870269fff0a1492e9a170850a3981347 [id=d4438b648a964ba6841ff82623ee18fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.760458 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c200681d4d77425389558e6489fb6032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.753010 1170 tablet_bootstrap.cc:654] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.766863 742 tablet_service.cc:1468] Processing CreateTablet for tablet cea5667d882b4b57a6fa4e9866641bf6 (DEFAULT_TABLE table=default.Table_ce82d3ecb4894b59aa7e82b457011d4a [id=68581ab2921a42d1931a9edb44fa1ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.767275 751 tablet_service.cc:1468] Processing CreateTablet for tablet ed8dae95b500437d869548995d84451f (DEFAULT_TABLE table=default.Table_596eb1526ec44e39bc70a926d0ed1d0c [id=111c965234a04f1c8b49f16fde760f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.767428 753 tablet_service.cc:1468] Processing CreateTablet for tablet 2edf591963ca46369c29a3fb2bf24bc3 (DEFAULT_TABLE table=default.Table_a0b7f5e829264dc5974e4d1039940a52 [id=c683387027d54cca98217b7398feed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.776671 3557 ts_tablet_manager.cc:1397] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.051s user 0.002s sys 0.001s
I20250905 19:52:41.777117 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9ae8fd5fdc4b539308268ba9729cd6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.778036 908 raft_consensus.cc:2466] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.778219 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:41.778270 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.778434 909 raft_consensus.cc:2466] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.778728 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:41.778904 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea5667d882b4b57a6fa4e9866641bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.778964 3557 raft_consensus.cc:357] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.779114 3557 raft_consensus.cc:383] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.779166 3557 raft_consensus.cc:738] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.779343 3557 consensus_queue.cc:260] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.779811 3557 ts_tablet_manager.cc:1428] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:41.781939 3151 consensus_queue.cc:1035] T c8e53152ada84954b12bbfe9d1fe3c16 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.783483 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:41.784276 2405 catalog_manager.cc:5582] T c8e53152ada84954b12bbfe9d1fe3c16 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.785241 2405 catalog_manager.cc:5582] T 5cbd471695e748c9831f9b0f6795aea7 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.785883 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
W20250905 19:52:41.785877 984 leader_election.cc:343] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.786311 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:41.786758 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:41.787142 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:41.787755 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:41.788276 2405 catalog_manager.cc:5582] T f34f0b29d6104cac8ab35309fd1edae8 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 2, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 2 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.788898 2405 catalog_manager.cc:5582] T 414f30a772524c33a74864ff741c865f P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } }
I20250905 19:52:41.789590 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:41.790304 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:41.790576 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:41.790670 3557 tablet_bootstrap.cc:492] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:41.790920 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:41.791239 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:41.792857 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:41.794075 2405 catalog_manager.cc:5582] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.794840 2405 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
W20250905 19:52:41.795483 984 leader_election.cc:343] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:41.795799 984 leader_election.cc:343] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.802505 986 leader_election.cc:304] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:41.802820 3297 raft_consensus.cc:2802] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:41.802943 3297 raft_consensus.cc:695] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:41.803166 3297 consensus_queue.cc:237] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.803951 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed8dae95b500437d869548995d84451f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.808176 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2edf591963ca46369c29a3fb2bf24bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.808238 3151 consensus_queue.cc:1035] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:41.814702 984 leader_election.cc:343] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.815701 2931 consensus_queue.cc:1035] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.816365 2931 consensus_queue.cc:1035] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:41.816931 984 leader_election.cc:343] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:41.817137 984 leader_election.cc:343] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.819244 3557 tablet_bootstrap.cc:654] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.822516 744 tablet_service.cc:1468] Processing CreateTablet for tablet ca00bfe551a34ca9bb6d14b3a24596b6 (DEFAULT_TABLE table=default.Table_124862b3c64045b4bc73ab2cb73400fa [id=0780adce3cb44cbfb8b20d8958a78fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.822974 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca00bfe551a34ca9bb6d14b3a24596b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.743893 3539 ts_tablet_manager.cc:1918] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:41.824060 3539 log.cc:1199] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9a81376d4c0b47a9817deae4d34ffb47
I20250905 19:52:41.824430 3539 ts_tablet_manager.cc:1939] T 9a81376d4c0b47a9817deae4d34ffb47 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:41.824828 3539 tablet_replica.cc:331] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:41.825029 3539 raft_consensus.cc:2241] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.825199 3539 raft_consensus.cc:2270] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.825477 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d1d24868204ce09a7f5e40357fec18" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:41.825706 3539 ts_tablet_manager.cc:1905] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.825733 905 raft_consensus.cc:2466] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:41.828315 739 tablet_service.cc:1468] Processing CreateTablet for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (DEFAULT_TABLE table=default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.828698 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e641ac5b5c49fda12450bbf1fec2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.829975 743 tablet_service.cc:1468] Processing CreateTablet for tablet 97912167c32646d18f005a9039d6eca6 (DEFAULT_TABLE table=default.Table_f779bfa94c9a4420ab2ec42650434ac1 [id=e9905045edce4aa9a8b17f2fa78ada25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.830327 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97912167c32646d18f005a9039d6eca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.832267 738 tablet_service.cc:1468] Processing CreateTablet for tablet 350197e134c549198771f4f8337d5dfc (DEFAULT_TABLE table=default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.832561 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350197e134c549198771f4f8337d5dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.833432 894 tablet_service.cc:1468] Processing CreateTablet for tablet 32934e708cff4efa9b1189b9eecada2c (DEFAULT_TABLE table=default.Table_8b15bc378227402a9b804f9e9f13c83e [id=3fdf4c4470564429aab8554dd5687d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.833904 877 tablet_service.cc:1468] Processing CreateTablet for tablet 97912167c32646d18f005a9039d6eca6 (DEFAULT_TABLE table=default.Table_f779bfa94c9a4420ab2ec42650434ac1 [id=e9905045edce4aa9a8b17f2fa78ada25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.834322 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32934e708cff4efa9b1189b9eecada2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.834784 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97912167c32646d18f005a9039d6eca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.835439 881 tablet_service.cc:1468] Processing CreateTablet for tablet ca00bfe551a34ca9bb6d14b3a24596b6 (DEFAULT_TABLE table=default.Table_124862b3c64045b4bc73ab2cb73400fa [id=0780adce3cb44cbfb8b20d8958a78fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.835758 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca00bfe551a34ca9bb6d14b3a24596b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.835839 745 tablet_service.cc:1468] Processing CreateTablet for tablet 1441704544e94844b0a262235318f790 (DEFAULT_TABLE table=default.Table_5462efe018cb409ea03a08a7b129264a [id=19d7086566de4c189c2a2392d775bf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.836172 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1441704544e94844b0a262235318f790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.836199 892 tablet_service.cc:1468] Processing CreateTablet for tablet 26905425077e4746bccb51431da706c9 (DEFAULT_TABLE table=default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.839905 875 tablet_service.cc:1468] Processing CreateTablet for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (DEFAULT_TABLE table=default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.840276 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e641ac5b5c49fda12450bbf1fec2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.840408 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26905425077e4746bccb51431da706c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.842116 751 tablet_service.cc:1468] Processing CreateTablet for tablet 26905425077e4746bccb51431da706c9 (DEFAULT_TABLE table=default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.842451 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26905425077e4746bccb51431da706c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.844208 883 tablet_service.cc:1468] Processing CreateTablet for tablet 350197e134c549198771f4f8337d5dfc (DEFAULT_TABLE table=default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.844537 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350197e134c549198771f4f8337d5dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.846765 893 tablet_service.cc:1468] Processing CreateTablet for tablet ed8dae95b500437d869548995d84451f (DEFAULT_TABLE table=default.Table_596eb1526ec44e39bc70a926d0ed1d0c [id=111c965234a04f1c8b49f16fde760f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.847067 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed8dae95b500437d869548995d84451f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.848389 3541 ts_tablet_manager.cc:1918] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:41.848522 3541 log.cc:1199] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9a81376d4c0b47a9817deae4d34ffb47
I20250905 19:52:41.848774 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]) successfully deleted
I20250905 19:52:41.849823 3541 ts_tablet_manager.cc:1939] T 9a81376d4c0b47a9817deae4d34ffb47 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:41.858588 753 tablet_service.cc:1468] Processing CreateTablet for tablet 32934e708cff4efa9b1189b9eecada2c (DEFAULT_TABLE table=default.Table_8b15bc378227402a9b804f9e9f13c83e [id=3fdf4c4470564429aab8554dd5687d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.859115 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32934e708cff4efa9b1189b9eecada2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.860896 1169 tablet_bootstrap.cc:492] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:41.861049 1169 ts_tablet_manager.cc:1397] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.299s user 0.008s sys 0.000s
I20250905 19:52:41.862344 3542 raft_consensus.cc:491] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.862572 3542 raft_consensus.cc:513] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.863554 3542 leader_election.cc:290] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:41.865557 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
W20250905 19:52:41.868340 984 leader_election.cc:343] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:41.868672 3274 consensus_queue.cc:1035] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.869063 3274 consensus_queue.cc:1035] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.869490 2373 catalog_manager.cc:5582] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.870464 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:41.870817 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:41.871234 1169 raft_consensus.cc:357] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.871393 1169 raft_consensus.cc:383] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.871452 1169 raft_consensus.cc:738] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.871680 1169 consensus_queue.cc:260] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.871620 2373 catalog_manager.cc:5582] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.872849 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:41.872997 1169 ts_tablet_manager.cc:1428] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.012s user 0.002s sys 0.000s
I20250905 19:52:41.873883 1169 tablet_bootstrap.cc:492] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:41.878225 2373 catalog_manager.cc:5582] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.878631 3274 raft_consensus.cc:491] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:41.878990 2373 catalog_manager.cc:5582] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.879858 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:41.880168 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:41.880743 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:41.881227 2373 catalog_manager.cc:5582] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.882215 2373 catalog_manager.cc:5582] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.882989 769 raft_consensus.cc:3058] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.882869 2373 catalog_manager.cc:5582] T 7ef9c9c6f81c4daea9008ada454b0659 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.883785 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:41.885273 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:41.886044 3274 raft_consensus.cc:513] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:41.885936 2373 catalog_manager.cc:5582] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.886512 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:41.886881 3274 leader_election.cc:290] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:41.886953 2373 catalog_manager.cc:5582] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.887406 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:41.887496 3547 ts_tablet_manager.cc:1918] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:41.887637 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:41.887837 3547 log.cc:1199] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/be1315824fd749049a62af3e6a113b9a
I20250905 19:52:41.888295 3547 ts_tablet_manager.cc:1939] T be1315824fd749049a62af3e6a113b9a P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:41.888500 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:41.889015 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:41.889279 3547 tablet_replica.cc:331] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:41.889470 2373 catalog_manager.cc:5582] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.890527 2373 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:41.890931 1026 tablet_service.cc:1468] Processing CreateTablet for tablet c200681d4d77425389558e6489fb6032 (DEFAULT_TABLE table=default.Table_870269fff0a1492e9a170850a3981347 [id=d4438b648a964ba6841ff82623ee18fb]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.890961 2373 catalog_manager.cc:5582] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:41.891438 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c200681d4d77425389558e6489fb6032. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.891726 2373 catalog_manager.cc:5582] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:41.892807 2397 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34736 (request call id 39) took 1054 ms. Trace:
I20250905 19:52:41.892948 2397 rpcz_store.cc:276] 0905 19:52:40.838355 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.242503 (+404148us) service_pool.cc:225] Handling call
0905 19:52:41.247719 (+ 5216us) spinlock_profiling.cc:243] Waited 4.98 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1bf4a7d 00007f73d1d80146 00007f73c8bb9126 00007f73c8bb8e55 00007f73c538d0be 00007f73c538bf9e 00007f73c53921d5 00007f73c5392aff
0905 19:52:41.248146 (+ 427us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.248416 (+ 270us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:41.248645 (+ 229us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:41.248725 (+ 80us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:41.340863 (+ 92138us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:41.892765 (+551902us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.299421 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.299535 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.299540 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:41.299542 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:41.299637 (+ 95us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.299650 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.299654 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:41.299713 (+ 59us) tablet.cc:699] Row locks acquired
0905 19:52:41.299715 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:41.299743 (+ 28us) write_op.cc:270] Start()
0905 19:52:41.299771 (+ 28us) write_op.cc:276] Timestamp: P: 1757101961299672 usec, L: 0
0905 19:52:41.299774 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.299946 (+ 172us) log.cc:844] Serialized 405 byte log entry
0905 19:52:41.340045 (+ 40099us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.340285 (+ 240us) write_op.cc:301] APPLY: starting
0905 19:52:41.340307 (+ 22us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.340363 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.340396 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.340461 (+ 65us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.340507 (+ 46us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:41.340519 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:41.340615 (+ 96us) log.cc:844] Serialized 23 byte log entry
0905 19:52:41.340683 (+ 68us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.340710 (+ 27us) write_op.cc:454] Released schema lock
0905 19:52:41.340719 (+ 9us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.354759 (+ 14040us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.354772 (+ 13us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":11443584,"child_traces":[["op",{"apply.queue_time_us":178,"apply.run_cpu_time_us":504,"apply.run_wall_time_us":504,"num_ops":1,"prepare.queue_time_us":50066,"prepare.run_cpu_time_us":752,"prepare.run_wall_time_us":44625,"raft.queue_time_us":1317,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":321,"replication_time_us":40229}]]}
I20250905 19:52:41.894178 2397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_d0a84be80bac45aa83ebf84bf8b0e08c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.894815 2397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d0a84be80bac45aa83ebf84bf8b0e08c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.899955 1169 tablet_bootstrap.cc:654] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:41.902251 1026 tablet_service.cc:1468] Processing CreateTablet for tablet 2edf591963ca46369c29a3fb2bf24bc3 (DEFAULT_TABLE table=default.Table_a0b7f5e829264dc5974e4d1039940a52 [id=c683387027d54cca98217b7398feed38]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.902724 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2edf591963ca46369c29a3fb2bf24bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.905084 1019 tablet_service.cc:1468] Processing CreateTablet for tablet cea5667d882b4b57a6fa4e9866641bf6 (DEFAULT_TABLE table=default.Table_ce82d3ecb4894b59aa7e82b457011d4a [id=68581ab2921a42d1931a9edb44fa1ecd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.907618 2397 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6016725fc2e44e0381a7cc4e06bcac86" } new_table_name: "default.Table_7ccec4c81ed34b1b9fa8ae78c4a47b9d" modify_external_catalogs: true
I20250905 19:52:41.912238 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet cea5667d882b4b57a6fa4e9866641bf6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.914458 3597 tablet.cc:1722] T e63b27ff4e434d76a8280f90c22f4082 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:41.916981 3557 tablet_bootstrap.cc:492] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:41.917402 2340 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 350197e134c549198771f4f8337d5dfc consensus state has no leader
I20250905 19:52:41.917578 2340 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34736 (request call id 33) took 1079 ms. Trace:
I20250905 19:52:41.917658 2340 rpcz_store.cc:276] 0905 19:52:40.837901 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.057224 (+219323us) service_pool.cc:225] Handling call
0905 19:52:41.058744 (+ 1520us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.058935 (+ 191us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:41.059078 (+ 143us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:41.059152 (+ 74us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:41.223529 (+164377us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:41.917525 (+693996us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.214390 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.214515 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.214520 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:41.214522 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:41.214618 (+ 96us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.214630 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.214635 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:41.214693 (+ 58us) tablet.cc:699] Row locks acquired
0905 19:52:41.214695 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:41.214721 (+ 26us) write_op.cc:270] Start()
0905 19:52:41.214748 (+ 27us) write_op.cc:276] Timestamp: P: 1757101961214649 usec, L: 0
0905 19:52:41.214751 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.214930 (+ 179us) log.cc:844] Serialized 405 byte log entry
0905 19:52:41.222551 (+ 7621us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.222801 (+ 250us) write_op.cc:301] APPLY: starting
0905 19:52:41.222822 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.222891 (+ 69us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.222937 (+ 46us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.223008 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.223056 (+ 48us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:41.223068 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:41.223163 (+ 95us) log.cc:844] Serialized 23 byte log entry
0905 19:52:41.223218 (+ 55us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.223238 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:41.223244 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.225607 (+ 2363us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.225613 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":116864,"child_traces":[["op",{"apply.queue_time_us":189,"apply.run_cpu_time_us":503,"apply.run_wall_time_us":503,"num_ops":1,"prepare.queue_time_us":154782,"prepare.run_cpu_time_us":748,"prepare.run_wall_time_us":11206,"raft.queue_time_us":82,"raft.run_cpu_time_us":354,"raft.run_wall_time_us":355,"replication_time_us":7759}]]}
I20250905 19:52:41.918565 2340 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_79e8c67d212c4e60be11fa071fe45c65"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.919122 2340 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79e8c67d212c4e60be11fa071fe45c65 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.919690 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 32934e708cff4efa9b1189b9eecada2c (DEFAULT_TABLE table=default.Table_8b15bc378227402a9b804f9e9f13c83e [id=3fdf4c4470564429aab8554dd5687d19]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.919917 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 97912167c32646d18f005a9039d6eca6 (DEFAULT_TABLE table=default.Table_f779bfa94c9a4420ab2ec42650434ac1 [id=e9905045edce4aa9a8b17f2fa78ada25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.920135 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 32934e708cff4efa9b1189b9eecada2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.920209 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ca00bfe551a34ca9bb6d14b3a24596b6 (DEFAULT_TABLE table=default.Table_124862b3c64045b4bc73ab2cb73400fa [id=0780adce3cb44cbfb8b20d8958a78fb9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.920317 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 97912167c32646d18f005a9039d6eca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.920668 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7ef9c9c6f81c4daea9008ada454b0659" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:41.921970 1019 tablet_service.cc:1468] Processing CreateTablet for tablet ed8dae95b500437d869548995d84451f (DEFAULT_TABLE table=default.Table_596eb1526ec44e39bc70a926d0ed1d0c [id=111c965234a04f1c8b49f16fde760f25]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.925297 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed8dae95b500437d869548995d84451f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.926664 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet be1315824fd749049a62af3e6a113b9a (table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]) successfully deleted
I20250905 19:52:41.926767 711 leader_election.cc:304] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:41.927018 3542 raft_consensus.cc:2747] T 7ef9c9c6f81c4daea9008ada454b0659 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:41.928905 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9a81376d4c0b47a9817deae4d34ffb47 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]) successfully deleted
I20250905 19:52:41.930073 751 tablet_service.cc:1468] Processing CreateTablet for tablet 92dae48543b944eca15e6cbd3066c7d3 (DEFAULT_TABLE table=default.Table_442ce59f4a4c4f1aac9b1c83d7535fbe [id=04007c80667049f4b18e09386248e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.930531 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92dae48543b944eca15e6cbd3066c7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.930742 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:41.931191 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 26905425077e4746bccb51431da706c9 (DEFAULT_TABLE table=default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.931314 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca00bfe551a34ca9bb6d14b3a24596b6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.931558 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26905425077e4746bccb51431da706c9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.932226 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 350197e134c549198771f4f8337d5dfc (DEFAULT_TABLE table=default.Table_ae712f1c45694ed9b3b777066df1d465 [id=248e84cad5c3471c947c275c89d6fe50]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.932544 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 350197e134c549198771f4f8337d5dfc. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.938246 3547 raft_consensus.cc:2241] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:41.938624 3547 raft_consensus.cc:2270] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.944161 3539 ts_tablet_manager.cc:1918] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:41.946648 2340 master_service.cc:948] ReplaceTablet: received request to replace tablet d7155cf5abba4e2cbdce583406f94fad from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:41.946931 2399 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34736 (request call id 34) took 1108 ms. Trace:
I20250905 19:52:41.947408 2399 rpcz_store.cc:276] 0905 19:52:40.837976 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.057723 (+219747us) service_pool.cc:225] Handling call
0905 19:52:41.058075 (+ 352us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.058299 (+ 224us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:41.058499 (+ 200us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:41.058575 (+ 76us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:41.212086 (+153511us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:41.946875 (+734789us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.206035 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.206220 (+ 185us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.206227 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:41.206231 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:41.206375 (+ 144us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.206395 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.206404 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:41.206489 (+ 85us) tablet.cc:699] Row locks acquired
0905 19:52:41.206493 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:41.206534 (+ 41us) write_op.cc:270] Start()
0905 19:52:41.206575 (+ 41us) write_op.cc:276] Timestamp: P: 1757101961206458 usec, L: 0
0905 19:52:41.206579 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.206761 (+ 182us) log.cc:844] Serialized 405 byte log entry
0905 19:52:41.211264 (+ 4503us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.211443 (+ 179us) write_op.cc:301] APPLY: starting
0905 19:52:41.211459 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.211502 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.211537 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.211595 (+ 58us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.211627 (+ 32us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:41.211634 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:41.211709 (+ 75us) log.cc:844] Serialized 23 byte log entry
0905 19:52:41.211981 (+ 272us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.212002 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:41.212008 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.214298 (+ 2290us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.214304 (+ 6us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":130,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":734,"num_ops":1,"prepare.queue_time_us":146953,"prepare.run_cpu_time_us":910,"prepare.run_wall_time_us":8260,"raft.queue_time_us":80,"raft.run_cpu_time_us":291,"raft.run_wall_time_us":291,"replication_time_us":4646}]]}
I20250905 19:52:41.948503 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_fe1d7268863740aea58d36b8bf9abc2c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:41.949159 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_fe1d7268863740aea58d36b8bf9abc2c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:41.952283 3539 log.cc:1199] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/89a73e984cbb4268b563f1ca01f4c65d
I20250905 19:52:41.957055 3539 ts_tablet_manager.cc:1939] T 89a73e984cbb4268b563f1ca01f4c65d P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:41.958802 3539 tablet_replica.cc:331] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:41.958982 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]) successfully deleted
I20250905 19:52:41.933158 1023 tablet_service.cc:1468] Processing CreateTablet for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (DEFAULT_TABLE table=default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.960090 3547 ts_tablet_manager.cc:1905] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.926452 875 tablet_service.cc:1468] Processing CreateTablet for tablet 1441704544e94844b0a262235318f790 (DEFAULT_TABLE table=default.Table_5462efe018cb409ea03a08a7b129264a [id=19d7086566de4c189c2a2392d775bf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.960567 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1441704544e94844b0a262235318f790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.927434 3541 tablet_replica.cc:331] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:41.929319 883 tablet_service.cc:1468] Processing CreateTablet for tablet 92dae48543b944eca15e6cbd3066c7d3 (DEFAULT_TABLE table=default.Table_442ce59f4a4c4f1aac9b1c83d7535fbe [id=04007c80667049f4b18e09386248e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:41.968583 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92dae48543b944eca15e6cbd3066c7d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.969982 3297 consensus_queue.cc:1035] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:41.968634 1170 tablet_bootstrap.cc:492] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:41.971058 3539 raft_consensus.cc:2241] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:41.971098 1170 ts_tablet_manager.cc:1397] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.231s user 0.005s sys 0.001s
I20250905 19:52:41.971266 3539 raft_consensus.cc:2270] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.971745 3539 ts_tablet_manager.cc:1905] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.971798 1170 raft_consensus.cc:357] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.971947 1170 raft_consensus.cc:383] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:41.972019 1170 raft_consensus.cc:738] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:41.972218 1170 consensus_queue.cc:260] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:41.973341 1170 ts_tablet_manager.cc:1428] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20250905 19:52:41.976465 3541 raft_consensus.cc:2241] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:41.977109 3541 raft_consensus.cc:2270] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:41.977507 986 leader_election.cc:304] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:41.977804 3541 ts_tablet_manager.cc:1905] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:41.978147 986 leader_election.cc:304] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:41.984696 1170 tablet_bootstrap.cc:492] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:41.985464 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c4e641ac5b5c49fda12450bbf1fec2b4. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:41.986927 2496 raft_consensus.cc:2802] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.987022 2496 raft_consensus.cc:491] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.987115 2496 raft_consensus.cc:3058] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.987471 3299 raft_consensus.cc:2802] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:41.987535 3299 raft_consensus.cc:491] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:41.987599 3299 raft_consensus.cc:3058] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:41.996687 2399 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_2c0cad91758c4851893e8a5919f5d44c" } modify_external_catalogs: true
I20250905 19:52:41.997430 2399 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_2c0cad91758c4851893e8a5919f5d44c" } modify_external_catalogs: true
I20250905 19:52:42.002653 3557 ts_tablet_manager.cc:1397] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.212s user 0.003s sys 0.000s
I20250905 19:52:42.003029 2392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 9f5d19f0216a4f069237e48c62ee7875
I20250905 19:52:42.003577 2392 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34736 (request call id 38) took 1165 ms. Trace:
I20250905 19:52:42.003685 2392 rpcz_store.cc:276] 0905 19:52:40.838264 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.165058 (+326794us) service_pool.cc:225] Handling call
0905 19:52:41.165558 (+ 500us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.165695 (+ 137us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:41.165779 (+ 84us) catalog_manager.cc:2799] Locking tablets
0905 19:52:41.256165 (+ 90386us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:41.457381 (+201216us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:41.457477 (+ 96us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:42.002940 (+545463us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:42.003542 (+ 602us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.354913 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.355051 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.355059 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:41.355063 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:41.355242 (+ 179us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.355262 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.355271 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:41.355419 (+ 148us) tablet.cc:699] Row locks acquired
0905 19:52:41.355423 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:41.355473 (+ 50us) write_op.cc:270] Start()
0905 19:52:41.355517 (+ 44us) write_op.cc:276] Timestamp: P: 1757101961355393 usec, L: 0
0905 19:52:41.355523 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.355783 (+ 260us) log.cc:844] Serialized 756 byte log entry
0905 19:52:41.456466 (+100683us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.456630 (+ 164us) write_op.cc:301] APPLY: starting
0905 19:52:41.456647 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.456695 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.456729 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.456799 (+ 70us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.456832 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:41.456839 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:41.456928 (+ 89us) log.cc:844] Serialized 29 byte log entry
0905 19:52:41.456986 (+ 58us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.457007 (+ 21us) write_op.cc:454] Released schema lock
0905 19:52:41.457013 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.457083 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.457087 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":117,"apply.run_cpu_time_us":444,"apply.run_wall_time_us":443,"num_ops":2,"prepare.queue_time_us":97964,"prepare.run_cpu_time_us":1110,"prepare.run_wall_time_us":56425,"raft.queue_time_us":3478,"raft.run_cpu_time_us":340,"raft.run_wall_time_us":340,"replication_time_us":100908}]]}
I20250905 19:52:42.004719 1170 tablet_bootstrap.cc:654] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.004750 2392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_18b5523c0ad64df4b15d6f0924914671"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.005502 2392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_18b5523c0ad64df4b15d6f0924914671 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.006318 3539 ts_tablet_manager.cc:1918] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.006422 3539 log.cc:1199] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/dcf448f9abc348df8d2d2501420d8f45
I20250905 19:52:42.006762 3539 ts_tablet_manager.cc:1939] T dcf448f9abc348df8d2d2501420d8f45 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.008555 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet dcf448f9abc348df8d2d2501420d8f45 (table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]) successfully deleted
W20250905 19:52:42.008874 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d709e2b4f10149a1818c1e1431b95937 because the tablet was not found. No further retry: Not found: Tablet not found: d709e2b4f10149a1818c1e1431b95937
I20250905 19:52:42.009397 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.009829 3299 raft_consensus.cc:513] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.010486 3299 leader_election.cc:290] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.012969 2496 raft_consensus.cc:513] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.013557 2496 leader_election.cc:290] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.015475 2392 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_ec95e016850b4e4eb1105ac761a31b74" } modify_external_catalogs: true
I20250905 19:52:42.015766 2392 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_ec95e016850b4e4eb1105ac761a31b74" } modify_external_catalogs: true
I20250905 19:52:42.028250 2392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6e3e1732bff14d49ade14fbfa6a9784c
I20250905 19:52:42.029706 2392 master_service.cc:948] ReplaceTablet: received request to replace tablet 6ee9f4fe3a464abaaca48f22ea183c0e from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:42.030699 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.031536 2342 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34736 (request call id 32) took 1193 ms. Trace:
I20250905 19:52:42.031663 2342 rpcz_store.cc:276] 0905 19:52:40.837817 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.057209 (+219392us) service_pool.cc:225] Handling call
0905 19:52:41.059742 (+ 2533us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.060021 (+ 279us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:41.060208 (+ 187us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:41.060301 (+ 93us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:41.234107 (+173806us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:42.031499 (+797392us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.225691 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.225810 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.225815 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:41.225818 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:41.225934 (+ 116us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.225947 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.225952 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:41.226016 (+ 64us) tablet.cc:699] Row locks acquired
0905 19:52:41.226018 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:41.226048 (+ 30us) write_op.cc:270] Start()
0905 19:52:41.226076 (+ 28us) write_op.cc:276] Timestamp: P: 1757101961225976 usec, L: 0
0905 19:52:41.226079 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.230370 (+ 4291us) log.cc:844] Serialized 405 byte log entry
0905 19:52:41.233501 (+ 3131us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.233671 (+ 170us) write_op.cc:301] APPLY: starting
0905 19:52:41.233686 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.233727 (+ 41us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.233752 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.233801 (+ 49us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.233830 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:41.233837 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:41.233917 (+ 80us) log.cc:844] Serialized 23 byte log entry
0905 19:52:41.233971 (+ 54us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.233990 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:41.233995 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.236294 (+ 2299us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.236304 (+ 10us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":1280,"child_traces":[["op",{"apply.queue_time_us":127,"apply.run_cpu_time_us":378,"apply.run_wall_time_us":378,"num_ops":1,"prepare.queue_time_us":164881,"prepare.run_cpu_time_us":761,"prepare.run_wall_time_us":10584,"raft.queue_time_us":64,"raft.run_cpu_time_us":236,"raft.run_wall_time_us":237,"replication_time_us":7392}]]}
I20250905 19:52:42.032756 2342 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_8ff658df694b488b9bea5ed9bae26cf8"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.033419 2342 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8ff658df694b488b9bea5ed9bae26cf8 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.039026 3615 tablet.cc:1722] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.039872 2370 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 087172093fff4c41980dc2daeabb07c3
I20250905 19:52:42.040093 1170 tablet_bootstrap.cc:492] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.040208 1170 ts_tablet_manager.cc:1397] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.056s user 0.001s sys 0.004s
I20250905 19:52:42.040520 2370 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34736 (request call id 31) took 1202 ms. Trace:
I20250905 19:52:42.040805 1170 raft_consensus.cc:357] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.040962 1170 raft_consensus.cc:383] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.041005 1170 raft_consensus.cc:738] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.041179 1170 consensus_queue.cc:260] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.040643 2370 rpcz_store.cc:276] 0905 19:52:40.837736 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.044995 (+207259us) service_pool.cc:225] Handling call
0905 19:52:41.053826 (+ 8831us) spinlock_profiling.cc:243] Waited 8.51 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1bf4a7d 00007f73d1d7eea6 00007f73c8bb7466 00007f73c8bb7195 00007f73c538d0be 00007f73c538bf9e 00007f73c53921d5 00007f73c5392aff
0905 19:52:41.054237 (+ 411us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.054358 (+ 121us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:41.054424 (+ 66us) catalog_manager.cc:2799] Locking tablets
0905 19:52:41.054559 (+ 135us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:41.202716 (+148157us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:41.202800 (+ 84us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:42.039792 (+836992us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:42.040482 (+ 690us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.169738 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.176026 (+ 6288us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.176037 (+ 11us) write_op.cc:435] Acquired schema lock
0905 19:52:41.176042 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:41.176880 (+ 838us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.176902 (+ 22us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.176911 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:41.185433 (+ 8522us) tablet.cc:699] Row locks acquired
0905 19:52:41.185439 (+ 6us) write_op.cc:260] PREPARE: finished
0905 19:52:41.185514 (+ 75us) write_op.cc:270] Start()
0905 19:52:41.185574 (+ 60us) write_op.cc:276] Timestamp: P: 1757101961185428 usec, L: 0
0905 19:52:41.185581 (+ 7us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.185886 (+ 305us) log.cc:844] Serialized 765 byte log entry
0905 19:52:41.198222 (+ 12336us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.198487 (+ 265us) write_op.cc:301] APPLY: starting
0905 19:52:41.198511 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.198577 (+ 66us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.198616 (+ 39us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.198714 (+ 98us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.198765 (+ 51us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:41.198778 (+ 13us) write_op.cc:312] APPLY: finished
0905 19:52:41.202302 (+ 3524us) log.cc:844] Serialized 29 byte log entry
0905 19:52:41.202605 (+ 303us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.202630 (+ 25us) write_op.cc:454] Released schema lock
0905 19:52:41.202638 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.205908 (+ 3270us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.205919 (+ 11us) write_op.cc:454] Released schema lock
Metrics: {"spinlock_wait_cycles":19563392,"child_traces":[["op",{"apply.queue_time_us":200,"apply.run_cpu_time_us":632,"apply.run_wall_time_us":4368,"num_ops":2,"prepare.queue_time_us":106440,"prepare.run_cpu_time_us":1504,"prepare.run_wall_time_us":44167,"raft.queue_time_us":74,"raft.run_cpu_time_us":284,"raft.run_wall_time_us":284,"replication_time_us":12594}]]}
I20250905 19:52:42.042624 1170 ts_tablet_manager.cc:1428] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:42.042752 3539 tablet_replica.cc:331] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.043651 2496 consensus_queue.cc:1035] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.043809 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 087172093fff4c41980dc2daeabb07c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.043875 2405 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_d6d822b5cec5446aa8b1db2a98074ef6 [id=d92c116575544c1c8d98befb5a420348] alter complete (version 2)
W20250905 19:52:42.044685 984 consensus_peers.cc:489] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:42.045058 2403 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_e892ac4352f44019ab940683425afff7 [id=ea980bda9f2f41c8bf82af3ce10d7151] alter complete (version 3)
I20250905 19:52:42.045564 3615 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:42.046974 1170 tablet_bootstrap.cc:492] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:42.047348 2397 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 1bd731a303a545f492b19f774520582c consensus state has no leader
I20250905 19:52:42.048403 1170 tablet_bootstrap.cc:654] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.049207 3539 raft_consensus.cc:2241] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.049319 3557 raft_consensus.cc:357] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.049651 3539 raft_consensus.cc:2270] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.049937 2372 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet e67ed7e9e5724de389504701584bd2b6
I20250905 19:52:42.050379 3539 ts_tablet_manager.cc:1905] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.052007 751 tablet_service.cc:1515] Processing DeleteTablet for tablet e67ed7e9e5724de389504701584bd2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.054126 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_d58a00eae8a14ac09495cf8262d3d13c"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.054718 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d58a00eae8a14ac09495cf8262d3d13c in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.056537 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_942a1f8066e34bfb9e2a520e352bd6d4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.057258 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_942a1f8066e34bfb9e2a520e352bd6d4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.058871 2397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_eae63ec4ceb6473bbd59b4a7c00e7e56"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.059548 2397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_eae63ec4ceb6473bbd59b4a7c00e7e56 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.068719 3542 raft_consensus.cc:491] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.068857 3542 raft_consensus.cc:513] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.069408 3542 leader_election.cc:290] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:42.070396 984 leader_election.cc:343] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.070490 2342 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_c76754fbc1b94a18a9695e46eec3d7a9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.070708 711 leader_election.cc:343] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: fd1280fefc3e43db9928567a734644f1
I20250905 19:52:42.070812 711 leader_election.cc:304] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:42.071058 3542 raft_consensus.cc:2747] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:42.071383 2342 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_c76754fbc1b94a18a9695e46eec3d7a9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.037474 3614 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:42.049453 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 087172093fff4c41980dc2daeabb07c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.049602 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.049707 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.078686 3557 raft_consensus.cc:383] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.078970 3557 raft_consensus.cc:738] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.079510 3557 consensus_queue.cc:260] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.080497 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.080657 908 raft_consensus.cc:2466] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:42.080744 3557 ts_tablet_manager.cc:1428] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.032s user 0.001s sys 0.000s
I20250905 19:52:42.080991 3557 tablet_bootstrap.cc:492] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.087718 3593 tablet.cc:1722] T e63b27ff4e434d76a8280f90c22f4082 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.068356 893 tablet_service.cc:1515] Processing DeleteTablet for tablet e67ed7e9e5724de389504701584bd2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.010565 3607 tablet.cc:1722] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.092350 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.101008 911 raft_consensus.cc:2391] T 539f67fac93c46ebb5897ea9f02016fd P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:42.107990 3542 raft_consensus.cc:491] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.108147 3542 raft_consensus.cc:513] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.108690 3542 leader_election.cc:290] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.112922 2403 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf] alter complete (version 1)
I20250905 19:52:42.113015 960 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:42.113407 1170 tablet_bootstrap.cc:492] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.113469 3610 raft_consensus.cc:491] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.113547 1170 ts_tablet_manager.cc:1397] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20250905 19:52:42.113613 3610 raft_consensus.cc:513] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.114264 3610 leader_election.cc:290] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.114257 1170 raft_consensus.cc:357] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.114411 1170 raft_consensus.cc:383] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.114460 1170 raft_consensus.cc:738] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.092388 3547 ts_tablet_manager.cc:1918] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.116078 3547 log.cc:1199] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/5a59d7fb5dc54b479d965282b1c7d24b
I20250905 19:52:42.116458 3547 ts_tablet_manager.cc:1939] T 5a59d7fb5dc54b479d965282b1c7d24b P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:42.100190 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 122064us
I20250905 19:52:42.122056 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:42.122783 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9ae8fd5fdc4b539308268ba9729cd6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:42.123061 768 raft_consensus.cc:3058] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.123181 3539 ts_tablet_manager.cc:1918] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.123399 3539 log.cc:1199] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d1a11777cf2d4ea9a5cdf792f73aff76
I20250905 19:52:42.123884 3539 ts_tablet_manager.cc:1939] T d1a11777cf2d4ea9a5cdf792f73aff76 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.124974 3539 tablet_replica.cc:331] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.125305 3539 raft_consensus.cc:2241] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.125532 3539 raft_consensus.cc:2270] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.125821 768 raft_consensus.cc:2466] T bd9ae8fd5fdc4b539308268ba9729cd6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.125881 1170 consensus_queue.cc:260] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.126370 1170 ts_tablet_manager.cc:1428] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20250905 19:52:42.126627 1170 tablet_bootstrap.cc:492] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.128257 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d1a11777cf2d4ea9a5cdf792f73aff76 (table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]) successfully deleted
I20250905 19:52:42.132063 1169 tablet_bootstrap.cc:492] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:42.132874 3557 tablet_bootstrap.cc:654] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.133174 1169 ts_tablet_manager.cc:1397] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.260s user 0.010s sys 0.009s
I20250905 19:52:42.137701 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.137917 911 raft_consensus.cc:2466] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:42.133430 3541 ts_tablet_manager.cc:1918] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:42.138429 3541 log.cc:1199] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/89a73e984cbb4268b563f1ca01f4c65d
I20250905 19:52:42.139163 3541 ts_tablet_manager.cc:1939] T 89a73e984cbb4268b563f1ca01f4c65d P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:42.142196 3539 ts_tablet_manager.cc:1905] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.148432 1170 tablet_bootstrap.cc:654] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.148743 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_789658eb5ab645dc90706bca2cd7eae9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.149433 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_789658eb5ab645dc90706bca2cd7eae9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.150029 3539 ts_tablet_manager.cc:1918] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.150126 3539 log.cc:1199] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/be1315824fd749049a62af3e6a113b9a
I20250905 19:52:42.150434 3539 ts_tablet_manager.cc:1939] T be1315824fd749049a62af3e6a113b9a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.151283 3539 tablet_replica.cc:331] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.151628 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet be1315824fd749049a62af3e6a113b9a (table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]) successfully deleted
I20250905 19:52:42.152050 3539 raft_consensus.cc:2241] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.152266 3539 raft_consensus.cc:2270] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.152801 3539 ts_tablet_manager.cc:1905] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.147451 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.147799 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bd9ae8fd5fdc4b539308268ba9729cd6" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.154865 906 raft_consensus.cc:3058] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.157660 906 raft_consensus.cc:2466] T bd9ae8fd5fdc4b539308268ba9729cd6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.150770 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.158032 907 raft_consensus.cc:2466] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:42.158257 905 raft_consensus.cc:3058] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.159112 3539 ts_tablet_manager.cc:1918] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.159337 3539 log.cc:1199] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/5a59d7fb5dc54b479d965282b1c7d24b
I20250905 19:52:42.159740 3539 ts_tablet_manager.cc:1939] T 5a59d7fb5dc54b479d965282b1c7d24b P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.160782 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 5a59d7fb5dc54b479d965282b1c7d24b (table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]) successfully deleted
I20250905 19:52:42.160769 905 raft_consensus.cc:2466] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.161434 3541 tablet_replica.cc:331] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:42.161572 2285 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9f5d19f0216a4f069237e48c62ee7875 because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 9f5d19f0216a4f069237e48c62ee7875 already in progress: creating tablet
I20250905 19:52:42.161703 3541 raft_consensus.cc:2241] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:42.161675 3536 consensus_queue.cc:1035] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:42.162256 2285 catalog_manager.cc:4908] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6e3e1732bff14d49ade14fbfa6a9784c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e3e1732bff14d49ade14fbfa6a9784c already in progress: creating tablet
I20250905 19:52:42.162524 3539 tablet_replica.cc:331] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.162490 1169 raft_consensus.cc:357] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.162652 1169 raft_consensus.cc:383] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.162721 1169 raft_consensus.cc:738] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.162863 3539 raft_consensus.cc:2241] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.162874 3536 raft_consensus.cc:491] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.162988 3536 raft_consensus.cc:513] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.163152 3539 raft_consensus.cc:2270] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.163528 3536 leader_election.cc:290] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.163787 3539 ts_tablet_manager.cc:1905] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.164490 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "539f67fac93c46ebb5897ea9f02016fd" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.164847 1170 tablet_bootstrap.cc:492] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.165030 1170 ts_tablet_manager.cc:1397] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.001s
I20250905 19:52:42.165027 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.165787 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 1441704544e94844b0a262235318f790 (DEFAULT_TABLE table=default.Table_5462efe018cb409ea03a08a7b129264a [id=19d7086566de4c189c2a2392d775bf36]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.165786 1170 raft_consensus.cc:357] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.166088 1170 raft_consensus.cc:383] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.166143 1170 raft_consensus.cc:738] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.166195 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1441704544e94844b0a262235318f790. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.166913 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 5a59d7fb5dc54b479d965282b1c7d24b (table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]) successfully deleted
I20250905 19:52:42.168231 2372 master_service.cc:948] ReplaceTablet: received request to replace tablet ad0760dfee934aacbd4ab1fb81e116c7 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:42.168247 1019 tablet_service.cc:1468] Processing CreateTablet for tablet 92dae48543b944eca15e6cbd3066c7d3 (DEFAULT_TABLE table=default.Table_442ce59f4a4c4f1aac9b1c83d7535fbe [id=04007c80667049f4b18e09386248e19c]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.168313 3610 consensus_queue.cc:1035] T 31484264112d42b089536d1ecc8ba021 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.168579 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 92dae48543b944eca15e6cbd3066c7d3. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:42.169890 984 leader_election.cc:343] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.172948 764 raft_consensus.cc:3058] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.174501 3541 raft_consensus.cc:2270] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.174881 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:42.175819 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:42.177307 711 leader_election.cc:304] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:42.178022 3346 raft_consensus.cc:2747] T 539f67fac93c46ebb5897ea9f02016fd P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:42.178980 773 raft_consensus.cc:3058] T cff0e7cc7c814627ad8964fa6a3b1576 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.166356 1170 consensus_queue.cc:260] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.181149 1170 ts_tablet_manager.cc:1428] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.016s user 0.001s sys 0.001s
I20250905 19:52:42.182301 2397 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:42.182473 1170 tablet_bootstrap.cc:492] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.185837 1170 tablet_bootstrap.cc:654] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.175128 3541 ts_tablet_manager.cc:1905] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.194633 3539 ts_tablet_manager.cc:1918] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250905 19:52:42.197443 3539 log.cc:1199] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/087172093fff4c41980dc2daeabb07c3
I20250905 19:52:42.198097 3539 ts_tablet_manager.cc:1939] T 087172093fff4c41980dc2daeabb07c3 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.199800 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]) successfully deleted
I20250905 19:52:42.198818 1170 tablet_bootstrap.cc:492] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.200737 1170 ts_tablet_manager.cc:1397] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.018s user 0.005s sys 0.001s
I20250905 19:52:42.201628 1170 raft_consensus.cc:357] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.201877 1170 raft_consensus.cc:383] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.201975 1170 raft_consensus.cc:738] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.202275 1170 consensus_queue.cc:260] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.203712 2342 master_service.cc:948] ReplaceTablet: received request to replace tablet b3ea2e2059a4425892fa3c0757e69e4c from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:42.203895 1170 ts_tablet_manager.cc:1428] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20250905 19:52:42.204941 1170 tablet_bootstrap.cc:492] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.206298 1170 tablet_bootstrap.cc:654] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.207011 3539 tablet_replica.cc:331] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.207415 3539 raft_consensus.cc:2241] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.207619 3539 raft_consensus.cc:2270] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.209491 3539 ts_tablet_manager.cc:1905] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.212184 3557 tablet_bootstrap.cc:492] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:42.212388 3557 ts_tablet_manager.cc:1397] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.131s user 0.001s sys 0.003s
I20250905 19:52:42.213173 3557 raft_consensus.cc:357] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.213424 3557 raft_consensus.cc:383] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.213513 3557 raft_consensus.cc:738] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.213745 3557 consensus_queue.cc:260] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.214412 3557 ts_tablet_manager.cc:1428] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:42.214710 3557 tablet_bootstrap.cc:492] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.216532 3557 tablet_bootstrap.cc:654] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.216833 1170 tablet_bootstrap.cc:492] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.217001 1170 ts_tablet_manager.cc:1397] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.000s
I20250905 19:52:42.217713 1170 raft_consensus.cc:357] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.218542 1170 raft_consensus.cc:383] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.218601 1170 raft_consensus.cc:738] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.218832 1170 consensus_queue.cc:260] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.220368 1170 ts_tablet_manager.cc:1428] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20250905 19:52:42.222734 1170 tablet_bootstrap.cc:492] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.224187 1170 tablet_bootstrap.cc:654] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.225313 3539 ts_tablet_manager.cc:1918] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.225474 3539 log.cc:1199] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e67ed7e9e5724de389504701584bd2b6
I20250905 19:52:42.225924 3539 ts_tablet_manager.cc:1939] T e67ed7e9e5724de389504701584bd2b6 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.227003 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet e67ed7e9e5724de389504701584bd2b6 (table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]) successfully deleted
I20250905 19:52:42.228379 1170 tablet_bootstrap.cc:492] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.228344 2405 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_fe3dee104cc640359210483bc31532cf [id=99384aac4e624d6db630b249c3ba2543] alter complete (version 3)
I20250905 19:52:42.228578 1170 ts_tablet_manager.cc:1397] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20250905 19:52:42.229301 1170 raft_consensus.cc:357] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.229663 1170 raft_consensus.cc:383] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.229758 1170 raft_consensus.cc:738] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.230044 1170 consensus_queue.cc:260] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.230612 2403 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_0d6fe65405f543d7a70643f94bf22da5 [id=46715848277847218d557157e4dfc77b] alter complete (version 1)
I20250905 19:52:42.231523 1170 ts_tablet_manager.cc:1428] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:42.232575 1170 tablet_bootstrap.cc:492] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.235165 712 leader_election.cc:304] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5
I20250905 19:52:42.235750 3346 raft_consensus.cc:2802] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:42.235819 3346 raft_consensus.cc:491] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4eb3af79b01d4835bf6bf7b26bf016f5)
I20250905 19:52:42.235891 3346 raft_consensus.cc:3058] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:42.238529 3346 raft_consensus.cc:513] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.239995 3346 leader_election.cc:290] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.240278 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 89a73e984cbb4268b563f1ca01f4c65d (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]) successfully deleted
I20250905 19:52:42.241693 1170 tablet_bootstrap.cc:654] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:42.252133 2292 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 350197e134c549198771f4f8337d5dfc (table default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431 [id=248e84cad5c3471c947c275c89d6fe50], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 350197e134c549198771f4f8337d5dfc consensus state has no leader
I20250905 19:52:42.170068 984 leader_election.cc:304] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:42.172196 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.172608 1029 tablet_service.cc:1515] Processing DeleteTablet for tablet 087172093fff4c41980dc2daeabb07c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.253310 3610 raft_consensus.cc:2802] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:42.172654 1028 tablet_service.cc:1515] Processing DeleteTablet for tablet e67ed7e9e5724de389504701584bd2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.253428 3610 raft_consensus.cc:695] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:42.253505 3547 tablet_replica.cc:331] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:42.253676 3610 consensus_queue.cc:237] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:42.233615 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 72209.7us
I20250905 19:52:42.254851 986 leader_election.cc:304] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.254949 2392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6ee9f4fe3a464abaaca48f22ea183c0e
I20250905 19:52:42.255399 3601 raft_consensus.cc:2802] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:42.255471 3601 raft_consensus.cc:491] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.255532 3601 raft_consensus.cc:3058] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.257619 751 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ee9f4fe3a464abaaca48f22ea183c0e with delete_type TABLET_DATA_DELETED (replaced by tablet b90713459cf14e49b75de34942fb13f0) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.257862 3539 tablet_replica.cc:331] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.258055 3539 raft_consensus.cc:2241] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.258260 2392 catalog_manager.cc:6652] ReplaceTablet: tablet 6ee9f4fe3a464abaaca48f22ea183c0e deleted and replaced by tablet b90713459cf14e49b75de34942fb13f0
I20250905 19:52:42.258193 3601 raft_consensus.cc:513] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.258286 3539 raft_consensus.cc:2270] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.258728 3601 leader_election.cc:290] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.258880 3601 consensus_queue.cc:1035] T e3cb1b098f42441594e28aeb1a9dbbf1 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.259232 3539 ts_tablet_manager.cc:1905] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.162890 1169 consensus_queue.cc:260] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.260005 1169 ts_tablet_manager.cc:1428] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.126s user 0.002s sys 0.000s
I20250905 19:52:42.260267 1169 tablet_bootstrap.cc:492] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:42.260707 3346 raft_consensus.cc:491] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.260838 3346 raft_consensus.cc:513] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.261422 3346 leader_election.cc:290] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.261610 1169 tablet_bootstrap.cc:654] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.262534 986 leader_election.cc:304] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.263113 3536 raft_consensus.cc:2802] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:42.263183 3536 raft_consensus.cc:491] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.263243 3536 raft_consensus.cc:3058] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.253681 3547 raft_consensus.cc:2241] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:42.264207 3547 pending_rounds.cc:70] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Trying to abort 1 pending ops.
I20250905 19:52:42.264248 3547 pending_rounds.cc:77] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089635547971584 op_type: NO_OP noop_request { }
I20250905 19:52:42.264322 3547 raft_consensus.cc:2887] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:42.264391 3547 raft_consensus.cc:2270] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.265980 3539 ts_tablet_manager.cc:1918] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.266177 3536 raft_consensus.cc:513] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.266752 3536 leader_election.cc:290] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.266976 3539 log.cc:1199] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6ee9f4fe3a464abaaca48f22ea183c0e
I20250905 19:52:42.236471 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.267488 3539 ts_tablet_manager.cc:1939] T 6ee9f4fe3a464abaaca48f22ea183c0e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.268014 3547 ts_tablet_manager.cc:1905] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.268904 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6ee9f4fe3a464abaaca48f22ea183c0e (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]) successfully deleted
I20250905 19:52:42.270054 3611 consensus_queue.cc:1035] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.270798 2931 consensus_queue.cc:1035] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.271102 2931 consensus_queue.cc:1035] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.271754 3611 consensus_queue.cc:1035] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.272199 2931 consensus_queue.cc:1035] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.279641 986 leader_election.cc:304] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:42.280030 3547 ts_tablet_manager.cc:1918] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.280208 3547 log.cc:1199] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/9f5d19f0216a4f069237e48c62ee7875
I20250905 19:52:42.280930 3547 ts_tablet_manager.cc:1939] T 9f5d19f0216a4f069237e48c62ee7875 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
W20250905 19:52:42.281829 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 83193.9us
I20250905 19:52:42.282353 2340 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d7155cf5abba4e2cbdce583406f94fad
I20250905 19:52:42.282797 3547 tablet_replica.cc:331] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:42.282894 2340 catalog_manager.cc:6652] ReplaceTablet: tablet d7155cf5abba4e2cbdce583406f94fad deleted and replaced by tablet 15be6e2e31364705beb09b4a636e3782
I20250905 19:52:42.283245 2340 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_0e8d769f23e9495cae82a20a516763f7" } new_table_name: "default.Table_34a19942c4604178a622037fc5303a1c" modify_external_catalogs: true
I20250905 19:52:42.283609 3547 raft_consensus.cc:2241] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.283818 1170 tablet_bootstrap.cc:492] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.283958 1170 ts_tablet_manager.cc:1397] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.051s user 0.004s sys 0.005s
I20250905 19:52:42.284176 3547 raft_consensus.cc:2270] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.284634 1170 raft_consensus.cc:357] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.284739 751 tablet_service.cc:1515] Processing DeleteTablet for tablet d7155cf5abba4e2cbdce583406f94fad with delete_type TABLET_DATA_DELETED (replaced by tablet 15be6e2e31364705beb09b4a636e3782) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.284801 1170 raft_consensus.cc:383] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.284886 1170 raft_consensus.cc:738] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.285096 1170 consensus_queue.cc:260] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.285080 3539 tablet_replica.cc:331] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.285482 3539 raft_consensus.cc:2241] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.285764 3547 ts_tablet_manager.cc:1905] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.286690 3539 raft_consensus.cc:2270] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.287070 893 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ee9f4fe3a464abaaca48f22ea183c0e with delete_type TABLET_DATA_DELETED (replaced by tablet b90713459cf14e49b75de34942fb13f0) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.287194 3539 ts_tablet_manager.cc:1905] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.287416 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d7155cf5abba4e2cbdce583406f94fad with delete_type TABLET_DATA_DELETED (replaced by tablet 15be6e2e31364705beb09b4a636e3782) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.287844 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.288100 908 raft_consensus.cc:3058] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:42.289604 1170 ts_tablet_manager.cc:1428] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.006s user 0.001s sys 0.000s
I20250905 19:52:42.290918 908 raft_consensus.cc:2466] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 2.
I20250905 19:52:42.294068 3539 ts_tablet_manager.cc:1918] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.294271 3539 log.cc:1199] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d7155cf5abba4e2cbdce583406f94fad
I20250905 19:52:42.294685 3539 ts_tablet_manager.cc:1939] T d7155cf5abba4e2cbdce583406f94fad P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.297278 3593 tablet.cc:1722] T e5f9e807244c4e0eb1f6dd1fc2acb9c1 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.303249 3541 ts_tablet_manager.cc:1918] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.280234 986 leader_election.cc:304] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.280064 3299 raft_consensus.cc:2802] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:42.303767 3299 raft_consensus.cc:491] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.303882 3299 raft_consensus.cc:3058] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.304688 3536 raft_consensus.cc:2802] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:42.304823 3536 raft_consensus.cc:695] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:42.305187 3536 consensus_queue.cc:237] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.306332 3299 raft_consensus.cc:513] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.307026 3299 leader_election.cc:290] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.307854 3541 log.cc:1199] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/dcf448f9abc348df8d2d2501420d8f45
I20250905 19:52:42.308422 3547 ts_tablet_manager.cc:1918] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250905 19:52:42.308523 3547 log.cc:1199] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/087172093fff4c41980dc2daeabb07c3
I20250905 19:52:42.308880 3547 ts_tablet_manager.cc:1939] T 087172093fff4c41980dc2daeabb07c3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:42.311020 3547 tablet_replica.cc:331] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:42.311240 3547 raft_consensus.cc:2241] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:42.311482 3547 raft_consensus.cc:2270] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.311981 3547 ts_tablet_manager.cc:1905] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.313443 3541 ts_tablet_manager.cc:1939] T dcf448f9abc348df8d2d2501420d8f45 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:42.283377 984 leader_election.cc:343] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.314595 3601 consensus_queue.cc:1035] T 539f67fac93c46ebb5897ea9f02016fd P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.315647 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 3) took 1017 ms. Trace:
I20250905 19:52:42.318295 3601 consensus_queue.cc:1035] T ad956757086b4d728698679c63c9b7ee P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.319444 2372 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet ad0760dfee934aacbd4ab1fb81e116c7
I20250905 19:52:42.320058 2372 catalog_manager.cc:6652] ReplaceTablet: tablet ad0760dfee934aacbd4ab1fb81e116c7 deleted and replaced by tablet 93694a3c232a48e180085cd490ee5f4d
I20250905 19:52:42.323130 751 tablet_service.cc:1515] Processing DeleteTablet for tablet ad0760dfee934aacbd4ab1fb81e116c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 93694a3c232a48e180085cd490ee5f4d) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.327409 3346 raft_consensus.cc:491] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.327576 3346 raft_consensus.cc:513] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.328473 3346 leader_election.cc:290] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.334383 1170 tablet_bootstrap.cc:492] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.321209 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.335558 907 raft_consensus.cc:2466] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:42.322304 3607 tablet.cc:1722] T e63b27ff4e434d76a8280f90c22f4082 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.336143 3547 ts_tablet_manager.cc:1918] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.336246 3547 log.cc:1199] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e67ed7e9e5724de389504701584bd2b6
I20250905 19:52:42.336305 1170 tablet_bootstrap.cc:654] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.336562 3547 ts_tablet_manager.cc:1939] T e67ed7e9e5724de389504701584bd2b6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:42.337745 3539 tablet_replica.cc:331] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.337931 3539 raft_consensus.cc:2241] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.338126 3539 raft_consensus.cc:2270] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.338625 3539 ts_tablet_manager.cc:1905] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.323380 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d1d24868204ce09a7f5e40357fec18" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.339959 905 raft_consensus.cc:3058] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.340781 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d7155cf5abba4e2cbdce583406f94fad (table default.Table_0e8d769f23e9495cae82a20a516763f7 [id=eed85872c9304732a9dfaa50c65ff816]) successfully deleted
I20250905 19:52:42.342554 905 raft_consensus.cc:2466] T 00d1d24868204ce09a7f5e40357fec18 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.343688 3579 raft_consensus.cc:491] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.343901 3579 raft_consensus.cc:513] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.346109 3579 leader_election.cc:290] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.346652 3539 ts_tablet_manager.cc:1918] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:42.346802 3539 log.cc:1199] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/ad0760dfee934aacbd4ab1fb81e116c7
I20250905 19:52:42.347214 3539 ts_tablet_manager.cc:1939] T ad0760dfee934aacbd4ab1fb81e116c7 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.347496 3579 raft_consensus.cc:491] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.347618 3579 raft_consensus.cc:513] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.348416 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:42.348595 3579 leader_election.cc:290] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.348845 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe00969eb9654976a0efeac2fa7c91d3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:42.349000 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.349184 905 raft_consensus.cc:2391] T c3b2fcc62ff0415f8073edb79b844600 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
W20250905 19:52:42.349444 848 leader_election.cc:343] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.349524 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5cf2bc19b043f68838f8a6af6393cd" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:42.349967 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.350095 1170 tablet_bootstrap.cc:492] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.350234 1170 ts_tablet_manager.cc:1397] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
W20250905 19:52:42.350286 848 leader_election.cc:343] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.350558 905 raft_consensus.cc:2391] T e77bdf9df6eb4470a3d22e96da952718 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:42.330797 3557 tablet_bootstrap.cc:492] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:42.350840 3557 ts_tablet_manager.cc:1397] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.136s user 0.005s sys 0.001s
I20250905 19:52:42.350953 1170 raft_consensus.cc:357] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.351115 1170 raft_consensus.cc:383] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.351166 1170 raft_consensus.cc:738] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.351424 1170 consensus_queue.cc:260] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.351593 3557 raft_consensus.cc:357] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.351738 3557 raft_consensus.cc:383] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.351812 3557 raft_consensus.cc:738] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.351953 1170 ts_tablet_manager.cc:1428] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:42.352242 1170 tablet_bootstrap.cc:492] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:42.352216 984 leader_election.cc:343] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.352816 712 leader_election.cc:304] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:42.353145 3346 raft_consensus.cc:2802] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:42.353272 3346 raft_consensus.cc:695] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:42.353535 3346 consensus_queue.cc:237] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.353999 1170 tablet_bootstrap.cc:654] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.356923 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5cf2bc19b043f68838f8a6af6393cd" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.357652 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe00969eb9654976a0efeac2fa7c91d3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.358237 3541 rpcz_store.cc:276] 0905 19:52:41.297699 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.298638 (+ 939us) service_pool.cc:225] Handling call
0905 19:52:42.161389 (+862751us) ts_tablet_manager.cc:1145] Deleting tablet dcf448f9abc348df8d2d2501420d8f45
0905 19:52:42.161396 (+ 7us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:42.193954 (+ 32558us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:42.303124 (+109170us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:42.315628 (+ 12504us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":862600}
I20250905 19:52:42.359239 3541 tablet_replica.cc:331] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:42.359491 3541 raft_consensus.cc:2241] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.362584 3557 consensus_queue.cc:260] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.364015 3601 consensus_queue.cc:1035] T cff0e7cc7c814627ad8964fa6a3b1576 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.364563 3557 ts_tablet_manager.cc:1428] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.014s user 0.001s sys 0.000s
I20250905 19:52:42.365788 775 raft_consensus.cc:3058] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.366150 769 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "00d1d24868204ce09a7f5e40357fec18" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:42.366298 769 raft_consensus.cc:3058] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.366999 3557 tablet_bootstrap.cc:492] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.368800 769 raft_consensus.cc:2466] T 00d1d24868204ce09a7f5e40357fec18 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.371800 3593 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:42.380790 1170 tablet_bootstrap.cc:492] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.380971 1170 ts_tablet_manager.cc:1397] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.029s user 0.005s sys 0.001s
I20250905 19:52:42.368889 3557 tablet_bootstrap.cc:654] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.381650 1170 raft_consensus.cc:357] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.381844 1170 raft_consensus.cc:383] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20250905 19:52:42.375916 849 leader_election.cc:343] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.381894 1170 raft_consensus.cc:738] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.382025 849 leader_election.cc:304] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.382079 1170 consensus_queue.cc:260] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.382668 1170 ts_tablet_manager.cc:1428] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:42.383484 3563 raft_consensus.cc:2747] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:42.384027 849 leader_election.cc:343] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.384305 849 leader_election.cc:304] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.384588 3563 raft_consensus.cc:2747] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:42.386368 893 tablet_service.cc:1515] Processing DeleteTablet for tablet ad0760dfee934aacbd4ab1fb81e116c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 93694a3c232a48e180085cd490ee5f4d) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.388358 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet dcf448f9abc348df8d2d2501420d8f45 (table default.Table_90a255f85d934941be4b07b8d27bdd8b [id=a2eb2d81960948e5be226c3bc32ff727]) successfully deleted
I20250905 19:52:42.388427 1170 tablet_bootstrap.cc:492] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:42.389012 2292 catalog_manager.cc:4895] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): delete failed for tablet d709e2b4f10149a1818c1e1431b95937 because the tablet was not found. No further retry: Not found: Tablet not found: d709e2b4f10149a1818c1e1431b95937
I20250905 19:52:42.390252 1170 tablet_bootstrap.cc:654] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.402006 908 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.402259 908 raft_consensus.cc:3058] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.404801 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:42.406124 1169 tablet_bootstrap.cc:492] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:42.406278 1169 ts_tablet_manager.cc:1397] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.146s user 0.004s sys 0.000s
I20250905 19:52:42.407102 1169 raft_consensus.cc:357] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.407305 1169 raft_consensus.cc:383] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.407374 1169 raft_consensus.cc:738] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.407583 1169 consensus_queue.cc:260] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.407745 3557 tablet_bootstrap.cc:492] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:42.408941 3557 ts_tablet_manager.cc:1397] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20250905 19:52:42.417862 1170 tablet_bootstrap.cc:492] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.419440 911 raft_consensus.cc:1238] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Rejecting Update request from peer 4eb3af79b01d4835bf6bf7b26bf016f5 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250905 19:52:42.419442 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]) successfully deleted
I20250905 19:52:42.420404 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:42.423000 3546 raft_consensus.cc:491] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.423159 3546 raft_consensus.cc:513] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.423802 3546 leader_election.cc:290] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.424823 3557 raft_consensus.cc:357] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.426232 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.427541 909 raft_consensus.cc:2391] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:42.427795 3557 raft_consensus.cc:383] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.428007 3557 raft_consensus.cc:738] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.428828 3557 consensus_queue.cc:260] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.430022 3557 ts_tablet_manager.cc:1428] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.020s user 0.001s sys 0.000s
I20250905 19:52:42.432551 3557 tablet_bootstrap.cc:492] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.436441 2372 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_5277901ee519486abb3c0e90d83961b5" } new_table_name: "default.Table_f055252e328e48ca9402b1265436d635" modify_external_catalogs: true
I20250905 19:52:42.436558 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]) successfully deleted
I20250905 19:52:42.437404 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e67ed7e9e5724de389504701584bd2b6 (table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]) successfully deleted
W20250905 19:52:42.437760 2291 catalog_manager.cc:4908] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): delete failed for tablet 6e3e1732bff14d49ade14fbfa6a9784c because tablet deleting was already in progress. No further retry: Already present: State transition of tablet 6e3e1732bff14d49ade14fbfa6a9784c already in progress: creating tablet
I20250905 19:52:42.438102 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_87c6aa6f867b47c693a3d535de50f2a2"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.438679 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_87c6aa6f867b47c693a3d535de50f2a2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.439802 3563 raft_consensus.cc:491] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.439929 3563 raft_consensus.cc:513] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.440573 3563 leader_election.cc:290] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.440861 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a95130c82e14bbcae577d4244b876ab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:42.441396 848 leader_election.cc:343] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.443400 3299 raft_consensus.cc:491] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.443579 3299 raft_consensus.cc:513] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.444200 3299 leader_election.cc:290] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.444149 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.444341 911 raft_consensus.cc:3058] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.447054 911 raft_consensus.cc:2466] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.453840 1170 ts_tablet_manager.cc:1397] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.065s user 0.018s sys 0.007s
I20250905 19:52:42.454515 1170 raft_consensus.cc:357] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.454681 1170 raft_consensus.cc:383] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.454727 1170 raft_consensus.cc:738] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.454909 1170 consensus_queue.cc:260] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.461160 908 raft_consensus.cc:2466] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.461627 3541 raft_consensus.cc:2270] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.462239 3541 ts_tablet_manager.cc:1905] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.467096 1169 ts_tablet_manager.cc:1428] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.061s user 0.002s sys 0.001s
I20250905 19:52:42.467458 1169 tablet_bootstrap.cc:492] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:42.468976 1169 tablet_bootstrap.cc:654] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.476578 1170 ts_tablet_manager.cc:1428] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.023s user 0.002s sys 0.001s
I20250905 19:52:42.476907 1170 tablet_bootstrap.cc:492] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.478252 1170 tablet_bootstrap.cc:654] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.479324 2342 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet b3ea2e2059a4425892fa3c0757e69e4c
I20250905 19:52:42.479897 2342 catalog_manager.cc:6652] ReplaceTablet: tablet b3ea2e2059a4425892fa3c0757e69e4c deleted and replaced by tablet fd769c4075914d338cbee14c97402aef
I20250905 19:52:42.480530 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ee9f4fe3a464abaaca48f22ea183c0e with delete_type TABLET_DATA_DELETED (replaced by tablet b90713459cf14e49b75de34942fb13f0) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.480778 3547 tablet_replica.cc:331] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:42.480981 3547 raft_consensus.cc:2241] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:42.481273 3547 raft_consensus.cc:2270] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.482235 3547 ts_tablet_manager.cc:1905] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.482272 751 tablet_service.cc:1515] Processing DeleteTablet for tablet b3ea2e2059a4425892fa3c0757e69e4c with delete_type TABLET_DATA_DELETED (replaced by tablet fd769c4075914d338cbee14c97402aef) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.482520 3539 tablet_replica.cc:331] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.482709 3539 raft_consensus.cc:2241] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.482888 3539 raft_consensus.cc:2270] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.483026 2342 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_5597507266624e81888c14b51a6adb97"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:42.483386 3539 ts_tablet_manager.cc:1905] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:42.483631 2342 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_5597507266624e81888c14b51a6adb97 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.490455 3539 ts_tablet_manager.cc:1918] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:42.490581 3539 log.cc:1199] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/b3ea2e2059a4425892fa3c0757e69e4c
I20250905 19:52:42.490916 3539 ts_tablet_manager.cc:1939] T b3ea2e2059a4425892fa3c0757e69e4c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.491509 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]) successfully deleted
I20250905 19:52:42.495237 3547 ts_tablet_manager.cc:1918] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.495466 3547 log.cc:1199] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/6ee9f4fe3a464abaaca48f22ea183c0e
I20250905 19:52:42.496150 3547 ts_tablet_manager.cc:1939] T 6ee9f4fe3a464abaaca48f22ea183c0e P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:42.501130 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet d7155cf5abba4e2cbdce583406f94fad with delete_type TABLET_DATA_DELETED (replaced by tablet 15be6e2e31364705beb09b4a636e3782) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.501399 1170 tablet_bootstrap.cc:492] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.501526 1170 ts_tablet_manager.cc:1397] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.025s user 0.004s sys 0.000s
I20250905 19:52:42.502154 1170 raft_consensus.cc:357] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.502305 1170 raft_consensus.cc:383] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.502346 1170 raft_consensus.cc:738] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.502557 1170 consensus_queue.cc:260] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.503136 1170 ts_tablet_manager.cc:1428] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:42.503417 1170 tablet_bootstrap.cc:492] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.504683 822 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:42.505016 1170 tablet_bootstrap.cc:654] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.512115 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet ad0760dfee934aacbd4ab1fb81e116c7 with delete_type TABLET_DATA_DELETED (replaced by tablet 93694a3c232a48e180085cd490ee5f4d) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.512656 1026 tablet_service.cc:1515] Processing DeleteTablet for tablet b3ea2e2059a4425892fa3c0757e69e4c with delete_type TABLET_DATA_DELETED (replaced by tablet fd769c4075914d338cbee14c97402aef) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:42.513896 751 tablet_service.cc:1468] Processing CreateTablet for tablet e2659c901336484fb3e4b1a892f05792 (DEFAULT_TABLE table=default.Table_d0a84be80bac45aa83ebf84bf8b0e08c [id=cc2a552eb2d9454eb462957f1892e007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.514309 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2659c901336484fb3e4b1a892f05792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.516258 743 tablet_service.cc:1468] Processing CreateTablet for tablet ed02bf5743f246a1937e9a9fe22c764e (DEFAULT_TABLE table=default.Table_fe6d663d69f74f929b1255e176be4ead [id=c8387d306f61429aba1dc7236167ded9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.516613 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed02bf5743f246a1937e9a9fe22c764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.516613 3601 consensus_queue.cc:1035] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.518486 986 leader_election.cc:304] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.518771 3299 raft_consensus.cc:2802] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:42.518842 3299 raft_consensus.cc:491] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.518916 3299 raft_consensus.cc:3058] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.520251 738 tablet_service.cc:1468] Processing CreateTablet for tablet 84a17cb780fe40e68277c89096a32aa0 (DEFAULT_TABLE table=default.Table_6ebce8ef802e4d9682e7815b99d7e8db [id=cc21cb8f8b704342a8a61be30eb22227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.520563 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a17cb780fe40e68277c89096a32aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.521725 3299 raft_consensus.cc:513] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.522302 3299 leader_election.cc:290] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.522650 986 leader_election.cc:304] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:42.523373 3547 tablet_replica.cc:331] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:42.523578 3547 raft_consensus.cc:2241] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:42.524251 753 tablet_service.cc:1468] Processing CreateTablet for tablet b2dd644e8d5d4b378e5239d43522ab07 (DEFAULT_TABLE table=default.Table_f1b2c338b96549e0b2fa23f258256b57 [id=96fb95c9bbd4456b8a2ea105b6db63d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.524341 1026 tablet_service.cc:1468] Processing CreateTablet for tablet e2659c901336484fb3e4b1a892f05792 (DEFAULT_TABLE table=default.Table_d0a84be80bac45aa83ebf84bf8b0e08c [id=cc2a552eb2d9454eb462957f1892e007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.524582 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dd644e8d5d4b378e5239d43522ab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.524857 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2659c901336484fb3e4b1a892f05792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.525254 2370 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_19e8169b84d0438ba2b732151d85cf52" } new_table_name: "default.Table_186206d66e7446e980e368d0716e67df" modify_external_catalogs: true
I20250905 19:52:42.528234 745 tablet_service.cc:1468] Processing CreateTablet for tablet 30925e81025445b58dd7c2563bb48da9 (DEFAULT_TABLE table=default.Table_3f0abddc641a42f590165ca4f321a535 [id=6e46eae4abf24c9db1c6eabe3b41a6e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.528548 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30925e81025445b58dd7c2563bb48da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.532006 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 84a17cb780fe40e68277c89096a32aa0 (DEFAULT_TABLE table=default.Table_6ebce8ef802e4d9682e7815b99d7e8db [id=cc21cb8f8b704342a8a61be30eb22227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.532354 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a17cb780fe40e68277c89096a32aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.534264 2342 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_e2abd597f41a4e4bb4c64bcba4bebd80"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.534824 2342 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e2abd597f41a4e4bb4c64bcba4bebd80 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:42.535768 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ed02bf5743f246a1937e9a9fe22c764e (DEFAULT_TABLE table=default.Table_fe6d663d69f74f929b1255e176be4ead [id=c8387d306f61429aba1dc7236167ded9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.536149 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed02bf5743f246a1937e9a9fe22c764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.536228 739 tablet_service.cc:1468] Processing CreateTablet for tablet 4990c72176624e44b7585ebf12581465 (DEFAULT_TABLE table=default.Table_b0e8d7317a7e41f7a3e73f23fe162bbe [id=2f3fd0ee16a640afab56e2dfd76dd92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.536545 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4990c72176624e44b7585ebf12581465. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.537390 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "96aaaf6d94594861b545f831f77dcb55" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 1 index: 1 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:42.538023 3299 raft_consensus.cc:2802] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:42.538152 3299 raft_consensus.cc:695] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:42.538465 3299 consensus_queue.cc:237] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.540179 744 tablet_service.cc:1468] Processing CreateTablet for tablet 823e311962dc40d3b700d8d6b4a70684 (DEFAULT_TABLE table=default.Table_3d26023ae0754b29bb4dfede839e5998 [id=504d611cb43b4eaf9504180fb02b00ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.540508 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823e311962dc40d3b700d8d6b4a70684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.544147 2342 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_a9dc5d6bf42c4c8f852069c46921e828" } modify_external_catalogs: true
I20250905 19:52:42.544329 2342 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_a9dc5d6bf42c4c8f852069c46921e828" } modify_external_catalogs: true
I20250905 19:52:42.544552 1029 tablet_service.cc:1468] Processing CreateTablet for tablet b2dd644e8d5d4b378e5239d43522ab07 (DEFAULT_TABLE table=default.Table_f1b2c338b96549e0b2fa23f258256b57 [id=96fb95c9bbd4456b8a2ea105b6db63d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.544925 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dd644e8d5d4b378e5239d43522ab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.546732 3547 raft_consensus.cc:2270] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.547320 3547 ts_tablet_manager.cc:1905] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.548215 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 30925e81025445b58dd7c2563bb48da9 (DEFAULT_TABLE table=default.Table_3f0abddc641a42f590165ca4f321a535 [id=6e46eae4abf24c9db1c6eabe3b41a6e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.548559 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30925e81025445b58dd7c2563bb48da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.552233 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 4990c72176624e44b7585ebf12581465 (DEFAULT_TABLE table=default.Table_b0e8d7317a7e41f7a3e73f23fe162bbe [id=2f3fd0ee16a640afab56e2dfd76dd92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.552583 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4990c72176624e44b7585ebf12581465. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.556288 1023 tablet_service.cc:1468] Processing CreateTablet for tablet 823e311962dc40d3b700d8d6b4a70684 (DEFAULT_TABLE table=default.Table_3d26023ae0754b29bb4dfede839e5998 [id=504d611cb43b4eaf9504180fb02b00ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.556622 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823e311962dc40d3b700d8d6b4a70684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.558651 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a95130c82e14bbcae577d4244b876ab" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.559931 1019 tablet_service.cc:1468] Processing CreateTablet for tablet b23111dc8af94b56af9538f973248f2a (DEFAULT_TABLE table=default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.560374 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b23111dc8af94b56af9538f973248f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.567857 893 tablet_service.cc:1515] Processing DeleteTablet for tablet b3ea2e2059a4425892fa3c0757e69e4c with delete_type TABLET_DATA_DELETED (replaced by tablet fd769c4075914d338cbee14c97402aef) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.568698 3557 tablet_bootstrap.cc:654] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.513896 742 tablet_service.cc:1468] Processing CreateTablet for tablet b23111dc8af94b56af9538f973248f2a (DEFAULT_TABLE table=default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.570308 893 tablet_service.cc:1468] Processing CreateTablet for tablet e2659c901336484fb3e4b1a892f05792 (DEFAULT_TABLE table=default.Table_d0a84be80bac45aa83ebf84bf8b0e08c [id=cc2a552eb2d9454eb462957f1892e007]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.570719 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e2659c901336484fb3e4b1a892f05792. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.570647 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c3b2fcc62ff0415f8073edb79b844600" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.570822 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b23111dc8af94b56af9538f973248f2a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.574941 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e77bdf9df6eb4470a3d22e96da952718" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.576094 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3742e3b46ed24b9082fa3ba5a8e5ee29" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.577800 893 tablet_service.cc:1468] Processing CreateTablet for tablet ed02bf5743f246a1937e9a9fe22c764e (DEFAULT_TABLE table=default.Table_fe6d663d69f74f929b1255e176be4ead [id=c8387d306f61429aba1dc7236167ded9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.578382 1169 tablet_bootstrap.cc:492] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:42.578414 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ed02bf5743f246a1937e9a9fe22c764e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.579587 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 6ee9f4fe3a464abaaca48f22ea183c0e (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]) successfully deleted
I20250905 19:52:42.579895 768 raft_consensus.cc:1238] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 LEADER]: Rejecting Update request from peer 4eb3af79b01d4835bf6bf7b26bf016f5 for earlier term 1. Current term is 2. Ops: []
I20250905 19:52:42.580246 765 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2b5ce70b784c13aa64053b2d1fe541" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:42.580425 765 raft_consensus.cc:2466] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:42.580919 768 raft_consensus.cc:3058] T 10fef848b9b746c5a0de6a8c3ca74f7a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.584426 883 tablet_service.cc:1468] Processing CreateTablet for tablet 84a17cb780fe40e68277c89096a32aa0 (DEFAULT_TABLE table=default.Table_6ebce8ef802e4d9682e7815b99d7e8db [id=cc21cb8f8b704342a8a61be30eb22227]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.584954 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 84a17cb780fe40e68277c89096a32aa0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.593298 883 tablet_service.cc:1468] Processing CreateTablet for tablet b2dd644e8d5d4b378e5239d43522ab07 (DEFAULT_TABLE table=default.Table_f1b2c338b96549e0b2fa23f258256b57 [id=96fb95c9bbd4456b8a2ea105b6db63d1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.593745 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2dd644e8d5d4b378e5239d43522ab07. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.598559 875 tablet_service.cc:1468] Processing CreateTablet for tablet 30925e81025445b58dd7c2563bb48da9 (DEFAULT_TABLE table=default.Table_3f0abddc641a42f590165ca4f321a535 [id=6e46eae4abf24c9db1c6eabe3b41a6e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.598953 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 30925e81025445b58dd7c2563bb48da9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.605515 892 tablet_service.cc:1468] Processing CreateTablet for tablet 823e311962dc40d3b700d8d6b4a70684 (DEFAULT_TABLE table=default.Table_3d26023ae0754b29bb4dfede839e5998 [id=504d611cb43b4eaf9504180fb02b00ec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.605944 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 823e311962dc40d3b700d8d6b4a70684. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.612277 875 tablet_service.cc:1468] Processing CreateTablet for tablet 4990c72176624e44b7585ebf12581465 (DEFAULT_TABLE table=default.Table_b0e8d7317a7e41f7a3e73f23fe162bbe [id=2f3fd0ee16a640afab56e2dfd76dd92e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.612758 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4990c72176624e44b7585ebf12581465. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.616568 892 tablet_service.cc:1468] Processing CreateTablet for tablet b23111dc8af94b56af9538f973248f2a (DEFAULT_TABLE table=default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.617017 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b23111dc8af94b56af9538f973248f2a. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:42.626276 849 leader_election.cc:343] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.626494 849 leader_election.cc:304] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:42.626811 3641 raft_consensus.cc:2747] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:42.631649 742 tablet_service.cc:1468] Processing CreateTablet for tablet 252afae7396d4d90b4647f4c4cc7b7cf (DEFAULT_TABLE table=default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.632064 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252afae7396d4d90b4647f4c4cc7b7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.632454 3546 raft_consensus.cc:491] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.632587 3546 raft_consensus.cc:513] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.633100 3546 leader_election.cc:290] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.633685 738 tablet_service.cc:1468] Processing CreateTablet for tablet ff347675042f49ec9f31e28df7dc690d (DEFAULT_TABLE table=default.Table_5cd8ba66cf734db5afca42d7fbf46281 [id=c9ff22868d3a499e9798155a4f1448a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.634013 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff347675042f49ec9f31e28df7dc690d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.636600 753 tablet_service.cc:1468] Processing CreateTablet for tablet bd3a516f3a364c9ba30fb58b1baed332 (DEFAULT_TABLE table=default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.637408 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3a516f3a364c9ba30fb58b1baed332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.637856 882 tablet_service.cc:1468] Processing CreateTablet for tablet bd3a516f3a364c9ba30fb58b1baed332 (DEFAULT_TABLE table=default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.637933 881 tablet_service.cc:1468] Processing CreateTablet for tablet 252afae7396d4d90b4647f4c4cc7b7cf (DEFAULT_TABLE table=default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.638257 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3a516f3a364c9ba30fb58b1baed332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.640273 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252afae7396d4d90b4647f4c4cc7b7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.580588 3601 consensus_queue.cc:1046] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.641875 3601 raft_consensus.cc:3053] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:52:42.641945 3601 raft_consensus.cc:738] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:42.642155 3601 consensus_queue.cc:260] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 1, Committed index: 1, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.642402 3601 raft_consensus.cc:3058] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:42.644451 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:42.636258 3593 tablet.cc:1722] T c77dd068ea5c495ca8aa0e7a6823d792 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.644819 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
W20250905 19:52:42.637580 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 44071.4us
I20250905 19:52:42.648902 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:42.650029 986 leader_election.cc:304] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:42.650312 1169 ts_tablet_manager.cc:1397] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.183s user 0.003s sys 0.000s
I20250905 19:52:42.650842 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.652056 3610 raft_consensus.cc:2802] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:42.652621 3610 raft_consensus.cc:695] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:42.652866 3610 consensus_queue.cc:237] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.654237 892 tablet_service.cc:1468] Processing CreateTablet for tablet ff347675042f49ec9f31e28df7dc690d (DEFAULT_TABLE table=default.Table_5cd8ba66cf734db5afca42d7fbf46281 [id=c9ff22868d3a499e9798155a4f1448a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.654711 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff347675042f49ec9f31e28df7dc690d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.659530 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:42.660178 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e63b27ff4e434d76a8280f90c22f4082 (table default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab]). Expected version 1 got 0
I20250905 19:52:42.660998 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:42.662096 766 raft_consensus.cc:3058] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.662745 3299 consensus_queue.cc:1035] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.662880 3557 tablet_bootstrap.cc:492] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:42.662989 3557 ts_tablet_manager.cc:1397] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.231s user 0.003s sys 0.000s
I20250905 19:52:42.663760 3557 raft_consensus.cc:357] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.663935 3557 raft_consensus.cc:383] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.664011 3557 raft_consensus.cc:738] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.664217 3593 tablet.cc:1722] T 1bd731a303a545f492b19f774520582c P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.664230 3557 consensus_queue.cc:260] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.669234 3557 ts_tablet_manager.cc:1428] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.006s user 0.000s sys 0.002s
I20250905 19:52:42.669653 3557 tablet_bootstrap.cc:492] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.676815 1169 raft_consensus.cc:357] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.677003 1169 raft_consensus.cc:383] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.677060 1169 raft_consensus.cc:738] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.677219 711 leader_election.cc:304] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:42.677295 1169 consensus_queue.cc:260] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.677476 3546 raft_consensus.cc:3058] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.677881 1169 ts_tablet_manager.cc:1428] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.027s user 0.002s sys 0.000s
I20250905 19:52:42.678174 1169 tablet_bootstrap.cc:492] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:42.680353 3546 raft_consensus.cc:2747] T c3b2fcc62ff0415f8073edb79b844600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:42.682523 3546 raft_consensus.cc:491] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.682642 3546 raft_consensus.cc:513] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.685248 3557 tablet_bootstrap.cc:654] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.683279 3546 leader_election.cc:290] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.690085 711 leader_election.cc:304] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:42.690630 3346 raft_consensus.cc:3058] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.690658 711 leader_election.cc:304] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:42.691093 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.691306 909 raft_consensus.cc:2391] T 2642bb409bb4473db034035b1b834739 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:42.693415 3346 raft_consensus.cc:2747] T e77bdf9df6eb4470a3d22e96da952718 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:42.693735 3346 raft_consensus.cc:3058] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.697018 3346 raft_consensus.cc:2747] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:42.700348 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:42.705264 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:42.707471 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 3 got 0
I20250905 19:52:42.708252 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:42.710482 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:42.711481 3546 raft_consensus.cc:491] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.711709 3546 raft_consensus.cc:513] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.712579 3546 leader_election.cc:290] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.712786 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.713045 909 raft_consensus.cc:2391] T 6b8bbbcac06944469e13f14ab1d80402 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:42.715991 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:42.720556 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:42.722127 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet c77dd068ea5c495ca8aa0e7a6823d792 (table default.Table_e08447fc15f5401eb931185a79ad62c8 [id=b3f6e906472a4471be3c9bdd0a25b96d]). Expected version 1 got 0
I20250905 19:52:42.722749 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:42.724393 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 350197e134c549198771f4f8337d5dfc (table default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431 [id=248e84cad5c3471c947c275c89d6fe50]). Expected version 1 got 0
I20250905 19:52:42.725055 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 1bd731a303a545f492b19f774520582c (table default.Table_7ccec4c81ed34b1b9fa8ae78c4a47b9d [id=ed7ed5bb46a84eb2bc77d1f948592244]). Expected version 1 got 0
W20250905 19:52:42.726133 848 consensus_peers.cc:489] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 087172093fff4c41980dc2daeabb07c3. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:42.727303 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:42.727870 1170 tablet_bootstrap.cc:492] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.728031 1170 ts_tablet_manager.cc:1397] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.225s user 0.003s sys 0.002s
I20250905 19:52:42.728662 1170 raft_consensus.cc:357] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.728832 1170 raft_consensus.cc:383] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.728920 1170 raft_consensus.cc:738] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.729173 1170 consensus_queue.cc:260] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.729239 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:42.729619 1170 ts_tablet_manager.cc:1428] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:42.729893 1170 tablet_bootstrap.cc:492] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.729861 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:42.731410 1170 tablet_bootstrap.cc:654] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.734335 2392 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:42.736488 3557 tablet_bootstrap.cc:492] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:42.736682 3557 ts_tablet_manager.cc:1397] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.067s user 0.005s sys 0.005s
I20250905 19:52:42.737460 3557 raft_consensus.cc:357] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.737689 3557 raft_consensus.cc:383] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.737785 3557 raft_consensus.cc:738] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.738827 3557 consensus_queue.cc:260] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.741173 3541 ts_tablet_manager.cc:1918] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.679797 1169 tablet_bootstrap.cc:654] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.704399 986 leader_election.cc:304] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
W20250905 19:52:42.740103 2291 consensus_peers.cc:489] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 -> Peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Couldn't send request to peer e38333af29db4bf6af41ee5a25d6701e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111). This is attempt 6: this message will repeat every 5th retry.
I20250905 19:52:42.708880 3601 consensus_queue.cc:1035] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.034s
I20250905 19:52:42.708101 3611 consensus_queue.cc:1035] T 5a5d38136c034b8497045176f749a98d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.036s
I20250905 19:52:42.748776 3601 raft_consensus.cc:2802] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:42.748965 3601 raft_consensus.cc:695] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:42.748976 1170 tablet_bootstrap.cc:492] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:42.749104 1170 ts_tablet_manager.cc:1397] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.019s user 0.002s sys 0.000s
I20250905 19:52:42.749322 3601 consensus_queue.cc:237] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.749742 1170 raft_consensus.cc:357] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.749923 1170 raft_consensus.cc:383] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.749982 1170 raft_consensus.cc:738] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.750169 1170 consensus_queue.cc:260] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.751902 1170 ts_tablet_manager.cc:1428] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
W20250905 19:52:42.752225 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 47852.2us
W20250905 19:52:42.753602 2340 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 15be6e2e31364705beb09b4a636e3782 has no consensus state
I20250905 19:52:42.754073 2340 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_58d960189f5d44cf9f5b6e8a7909461a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:42.754716 2340 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_58d960189f5d44cf9f5b6e8a7909461a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:42.709328 984 consensus_peers.cc:489] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: BOOTSTRAPPING. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:42.759249 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 49954us
I20250905 19:52:42.764390 2340 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_a98611b77e2d4e7c868fe9641069ae23" } modify_external_catalogs: true
I20250905 19:52:42.764632 2340 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_a98611b77e2d4e7c868fe9641069ae23" } modify_external_catalogs: true
I20250905 19:52:42.770661 3666 raft_consensus.cc:491] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.770805 3666 raft_consensus.cc:513] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.771337 3666 leader_election.cc:290] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:42.773752 712 consensus_peers.cc:489] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe -> Peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Couldn't send request to peer 835daf30e4324782848290be181ae78f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: edb9379335ca4eb5a32a6cbe85601a1f. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:42.778415 2397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb
I20250905 19:52:42.779036 2397 catalog_manager.cc:6652] ReplaceTablet: tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb deleted and replaced by tablet fe4cc17bdb6d4af8b323de5116aec179
I20250905 19:52:42.779485 2397 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_01c7456a16814955b2c2f3890ecb0d01" } new_table_name: "default.Table_5bd287e779f048a584c35879c6f87713" modify_external_catalogs: true
I20250905 19:52:42.783442 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb with delete_type TABLET_DATA_DELETED (replaced by tablet fe4cc17bdb6d4af8b323de5116aec179) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:42.789422 3678 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:42.799078 3666 raft_consensus.cc:491] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.799234 3666 raft_consensus.cc:513] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.799919 3666 leader_election.cc:290] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:42.837025 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 76641.4us
W20250905 19:52:42.840716 984 leader_election.cc:343] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:42.841797 984 leader_election.cc:343] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.847047 753 tablet_service.cc:1468] Processing CreateTablet for tablet 452872677922455d96126751a1b464ef (DEFAULT_TABLE table=default.Table_e2abd597f41a4e4bb4c64bcba4bebd80 [id=9cf2797afe514db487c3fbd79ddacab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.850860 3666 raft_consensus.cc:491] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4eb3af79b01d4835bf6bf7b26bf016f5)
I20250905 19:52:42.851035 3666 raft_consensus.cc:513] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.851709 3666 leader_election.cc:290] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.853749 3274 raft_consensus.cc:491] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.853874 3274 raft_consensus.cc:513] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.854364 3274 leader_election.cc:290] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.855592 3274 consensus_queue.cc:1035] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:42.862756 738 tablet_service.cc:1468] Processing CreateTablet for tablet 11ed17b2bdf649a3a13b9c8657945889 (DEFAULT_TABLE table=default.Table_87c6aa6f867b47c693a3d535de50f2a2 [id=af65dfe93b874c02b16368eff59e3ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.863884 751 tablet_service.cc:1468] Processing CreateTablet for tablet da588ad7d1874575a41533d298e8e2b7 (DEFAULT_TABLE table=default.Table_eae63ec4ceb6473bbd59b4a7c00e7e56 [id=088e6e4646fc4c218a4e433ce0f9fade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.868948 984 leader_election.cc:304] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:42.869555 3151 raft_consensus.cc:2802] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:42.869697 3151 raft_consensus.cc:491] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.869733 742 tablet_service.cc:1468] Processing CreateTablet for tablet ea1b6c96305e49b9ad025baa41ffae33 (DEFAULT_TABLE table=default.Table_942a1f8066e34bfb9e2a520e352bd6d4 [id=60f1cde785a24b4db748918ef5df5cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.869848 3151 raft_consensus.cc:3058] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.871400 744 tablet_service.cc:1468] Processing CreateTablet for tablet 29848b701d98421a81f163ffe100d8a7 (DEFAULT_TABLE table=default.Table_d58a00eae8a14ac09495cf8262d3d13c [id=1e7d0dbdca914d2ca76cc8eb85eccbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.872675 3151 raft_consensus.cc:513] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.873378 3151 leader_election.cc:290] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.873963 3666 raft_consensus.cc:491] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.874094 3666 raft_consensus.cc:513] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.874655 3666 leader_election.cc:290] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.876452 739 tablet_service.cc:1468] Processing CreateTablet for tablet 46e5941c519a43d7ac6ba93e471e7325 (DEFAULT_TABLE table=default.Table_fe1d7268863740aea58d36b8bf9abc2c [id=706dc1f46c1c487e8911167fcc5c4a76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.879922 752 tablet_service.cc:1468] Processing CreateTablet for tablet 15be6e2e31364705beb09b4a636e3782 (DEFAULT_TABLE table=default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.880496 743 tablet_service.cc:1468] Processing CreateTablet for tablet 68e2c8572d3a4569b90664ecc4be26f2 (DEFAULT_TABLE table=default.Table_79e8c67d212c4e60be11fa071fe45c65 [id=d5ab4a8df913481b91cedb10c7171146]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.880868 745 tablet_service.cc:1468] Processing CreateTablet for tablet 6b6ae48042fe4cf99222e3b3c30a0b3f (DEFAULT_TABLE table=default.Table_8ff658df694b488b9bea5ed9bae26cf8 [id=11666097a7d8460690a76a09928574f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.881264 1170 tablet_bootstrap.cc:492] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:42.881579 1169 tablet_bootstrap.cc:492] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:42.881743 1169 ts_tablet_manager.cc:1397] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.204s user 0.003s sys 0.000s
I20250905 19:52:42.882637 1169 raft_consensus.cc:357] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.882843 1169 raft_consensus.cc:383] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.882906 1169 raft_consensus.cc:738] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.883002 1170 tablet_bootstrap.cc:654] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.883158 1169 consensus_queue.cc:260] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.760686 3557 ts_tablet_manager.cc:1428] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.024s user 0.002s sys 0.000s
I20250905 19:52:42.883817 3557 tablet_bootstrap.cc:492] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.884161 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:42.884457 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:42.885275 3557 tablet_bootstrap.cc:654] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.885478 3539 tablet_replica.cc:331] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:42.885730 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452872677922455d96126751a1b464ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.763983 3563 raft_consensus.cc:491] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.885962 3563 raft_consensus.cc:513] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.886584 3563 leader_election.cc:290] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.766327 906 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2b5ce70b784c13aa64053b2d1fe541" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.887069 906 raft_consensus.cc:2466] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:42.768705 3607 tablet.cc:1722] T c77dd068ea5c495ca8aa0e7a6823d792 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.775584 914 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.887624 914 raft_consensus.cc:2466] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:42.786110 3579 raft_consensus.cc:491] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.887832 3579 raft_consensus.cc:513] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.888180 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ed17b2bdf649a3a13b9c8657945889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.888373 3579 leader_election.cc:290] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.789991 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.888662 907 raft_consensus.cc:3058] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.791651 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb with delete_type TABLET_DATA_DELETED (replaced by tablet fe4cc17bdb6d4af8b323de5116aec179) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:42.817103 3677 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:42.827574 3682 tablet.cc:1722] T 1bd731a303a545f492b19f774520582c P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.892295 749 tablet_service.cc:1468] Processing CreateTablet for tablet ac73a74ec58b4b0b9f34af938b841aa1 (DEFAULT_TABLE table=default.Table_5597507266624e81888c14b51a6adb97 [id=1e9e241158ed4a7c9d0a808a89bf8b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.892673 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac73a74ec58b4b0b9f34af938b841aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.895596 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da588ad7d1874575a41533d298e8e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.895781 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e5941c519a43d7ac6ba93e471e7325. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.896351 740 tablet_service.cc:1468] Processing CreateTablet for tablet 799c195c42184db6bab055b209e335d9 (DEFAULT_TABLE table=default.Table_c76754fbc1b94a18a9695e46eec3d7a9 [id=9878febd268641a08408e41ec155c569]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.896701 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799c195c42184db6bab055b209e335d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.898554 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29848b701d98421a81f163ffe100d8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.899690 907 raft_consensus.cc:2466] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:42.903743 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1b6c96305e49b9ad025baa41ffae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.904026 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:42.904529 848 leader_election.cc:343] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.906287 740 tablet_service.cc:1468] Processing CreateTablet for tablet ff81146b9cd44dcba49b56ae0244f985 (DEFAULT_TABLE table=default.Table_789658eb5ab645dc90706bca2cd7eae9 [id=b87cccf4f95f4e93959679b75166ecd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.906534 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:42.867372 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 75571.2us
I20250905 19:52:42.909626 741 tablet_service.cc:1468] Processing CreateTablet for tablet 73671be59bd248559fdcaf0ca2a30f16 (DEFAULT_TABLE table=default.Table_18b5523c0ad64df4b15d6f0924914671 [id=534cd54186e448f98488cc8b04e8f5ba]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:42.909754 848 leader_election.cc:343] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.910051 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73671be59bd248559fdcaf0ca2a30f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.912190 3593 tablet.cc:1722] T 544dffc8b77049c295921d0af091b17d P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:42.912462 3541 log.cc:1199] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/be1315824fd749049a62af3e6a113b9a
I20250905 19:52:42.912945 3541 ts_tablet_manager.cc:1939] T be1315824fd749049a62af3e6a113b9a P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:42.913717 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 17) took 1273 ms. Trace:
I20250905 19:52:42.913790 3541 rpcz_store.cc:276] 0905 19:52:41.640109 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.640177 (+ 68us) service_pool.cc:225] Handling call
0905 19:52:42.359194 (+719017us) ts_tablet_manager.cc:1145] Deleting tablet be1315824fd749049a62af3e6a113b9a
0905 19:52:42.359200 (+ 6us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:42.640045 (+280845us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:42.741063 (+101018us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:42.913697 (+172634us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":101925,"tablet-delete.queue_time_us":649065}
I20250905 19:52:42.914099 3541 tablet_replica.cc:331] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: stopping tablet replica
W20250905 19:52:42.914469 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 45231.5us
I20250905 19:52:42.915032 3547 ts_tablet_manager.cc:1918] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:42.915143 3547 log.cc:1199] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/d7155cf5abba4e2cbdce583406f94fad
I20250905 19:52:42.915486 3547 ts_tablet_manager.cc:1939] T d7155cf5abba4e2cbdce583406f94fad P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:42.915535 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e2c8572d3a4569b90664ecc4be26f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.916025 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b6ae48042fe4cf99222e3b3c30a0b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.916481 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15be6e2e31364705beb09b4a636e3782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.917053 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.917227 712 leader_election.cc:304] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:42.917464 3666 raft_consensus.cc:2802] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:42.917532 3666 raft_consensus.cc:491] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.917613 3666 raft_consensus.cc:3058] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.918254 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:42.920668 3666 raft_consensus.cc:513] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:42.921293 3666 leader_election.cc:290] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.925148 3539 raft_consensus.cc:2241] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.925329 3539 raft_consensus.cc:2270] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.925824 3539 ts_tablet_manager.cc:1905] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.927915 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.929456 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:42.929615 907 raft_consensus.cc:2466] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:42.930399 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:42.930584 907 raft_consensus.cc:2391] T 6edbc312fc4947a79e937dbc64d30b37 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:42.931136 892 tablet_service.cc:1468] Processing CreateTablet for tablet 452872677922455d96126751a1b464ef (DEFAULT_TABLE table=default.Table_e2abd597f41a4e4bb4c64bcba4bebd80 [id=9cf2797afe514db487c3fbd79ddacab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.931488 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452872677922455d96126751a1b464ef. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:42.932443 848 consensus_peers.cc:489] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b3ea2e2059a4425892fa3c0757e69e4c. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:42.932861 749 tablet_service.cc:1468] Processing CreateTablet for tablet fd769c4075914d338cbee14c97402aef (DEFAULT_TABLE table=default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.933185 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd769c4075914d338cbee14c97402aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.933523 2370 catalog_manager.cc:5582] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 1 to 2, leader changed from 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3) to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 2 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:42.934123 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:42.935050 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:42.935232 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:42.935547 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:42.937898 882 tablet_service.cc:1468] Processing CreateTablet for tablet da588ad7d1874575a41533d298e8e2b7 (DEFAULT_TABLE table=default.Table_eae63ec4ceb6473bbd59b4a7c00e7e56 [id=088e6e4646fc4c218a4e433ce0f9fade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.938220 875 tablet_service.cc:1468] Processing CreateTablet for tablet 46e5941c519a43d7ac6ba93e471e7325 (DEFAULT_TABLE table=default.Table_fe1d7268863740aea58d36b8bf9abc2c [id=706dc1f46c1c487e8911167fcc5c4a76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.938328 878 tablet_service.cc:1468] Processing CreateTablet for tablet 799c195c42184db6bab055b209e335d9 (DEFAULT_TABLE table=default.Table_c76754fbc1b94a18a9695e46eec3d7a9 [id=9878febd268641a08408e41ec155c569]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.938488 881 tablet_service.cc:1468] Processing CreateTablet for tablet ea1b6c96305e49b9ad025baa41ffae33 (DEFAULT_TABLE table=default.Table_942a1f8066e34bfb9e2a520e352bd6d4 [id=60f1cde785a24b4db748918ef5df5cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.938669 893 tablet_service.cc:1468] Processing CreateTablet for tablet 68e2c8572d3a4569b90664ecc4be26f2 (DEFAULT_TABLE table=default.Table_79e8c67d212c4e60be11fa071fe45c65 [id=d5ab4a8df913481b91cedb10c7171146]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.938755 891 tablet_service.cc:1468] Processing CreateTablet for tablet ff81146b9cd44dcba49b56ae0244f985 (DEFAULT_TABLE table=default.Table_789658eb5ab645dc90706bca2cd7eae9 [id=b87cccf4f95f4e93959679b75166ecd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.938907 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6b6ae48042fe4cf99222e3b3c30a0b3f (DEFAULT_TABLE table=default.Table_8ff658df694b488b9bea5ed9bae26cf8 [id=11666097a7d8460690a76a09928574f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.939119 889 tablet_service.cc:1468] Processing CreateTablet for tablet ac73a74ec58b4b0b9f34af938b841aa1 (DEFAULT_TABLE table=default.Table_5597507266624e81888c14b51a6adb97 [id=1e9e241158ed4a7c9d0a808a89bf8b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.939150 892 tablet_service.cc:1468] Processing CreateTablet for tablet 11ed17b2bdf649a3a13b9c8657945889 (DEFAULT_TABLE table=default.Table_87c6aa6f867b47c693a3d535de50f2a2 [id=af65dfe93b874c02b16368eff59e3ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.939330 894 tablet_service.cc:1468] Processing CreateTablet for tablet 15be6e2e31364705beb09b4a636e3782 (DEFAULT_TABLE table=default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:42.939584 712 leader_election.cc:343] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Not found: Tablet not found: 3c824f5591044c4198a51a25661f71e5
I20250905 19:52:42.939633 876 tablet_service.cc:1468] Processing CreateTablet for tablet fd769c4075914d338cbee14c97402aef (DEFAULT_TABLE table=default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.939785 886 tablet_service.cc:1468] Processing CreateTablet for tablet 73671be59bd248559fdcaf0ca2a30f16 (DEFAULT_TABLE table=default.Table_18b5523c0ad64df4b15d6f0924914671 [id=534cd54186e448f98488cc8b04e8f5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.939947 712 leader_election.cc:304] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:42.937932 883 tablet_service.cc:1468] Processing CreateTablet for tablet 29848b701d98421a81f163ffe100d8a7 (DEFAULT_TABLE table=default.Table_d58a00eae8a14ac09495cf8262d3d13c [id=1e7d0dbdca914d2ca76cc8eb85eccbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.940263 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet be1315824fd749049a62af3e6a113b9a (table default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]) successfully deleted
I20250905 19:52:42.938284 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da588ad7d1874575a41533d298e8e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.940496 3546 raft_consensus.cc:2802] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election won for term 2
I20250905 19:52:42.940564 3546 raft_consensus.cc:491] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:42.940619 3546 raft_consensus.cc:3058] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:42.940691 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff81146b9cd44dcba49b56ae0244f985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.940874 886 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73671be59bd248559fdcaf0ca2a30f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.941315 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e5941c519a43d7ac6ba93e471e7325. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.943193 3546 raft_consensus.cc:513] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.943641 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799c195c42184db6bab055b209e335d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.944063 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e2c8572d3a4569b90664ecc4be26f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.944324 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b6ae48042fe4cf99222e3b3c30a0b3f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.944531 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd769c4075914d338cbee14c97402aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.944692 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15be6e2e31364705beb09b4a636e3782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.944712 3546 leader_election.cc:290] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:42.945096 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ed17b2bdf649a3a13b9c8657945889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.948068 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1b6c96305e49b9ad025baa41ffae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.950491 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac73a74ec58b4b0b9f34af938b841aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.951398 3557 tablet_bootstrap.cc:492] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:42.951503 3557 ts_tablet_manager.cc:1397] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.003s
I20250905 19:52:42.952075 883 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29848b701d98421a81f163ffe100d8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.952230 3557 raft_consensus.cc:357] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.952389 3557 raft_consensus.cc:383] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:42.952435 3557 raft_consensus.cc:738] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:42.952627 3557 consensus_queue.cc:260] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.953099 3557 ts_tablet_manager.cc:1428] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:42.953347 3557 tablet_bootstrap.cc:492] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:42.955449 3557 tablet_bootstrap.cc:654] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.956311 3541 raft_consensus.cc:2241] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.956466 3541 raft_consensus.cc:2270] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.956944 3541 ts_tablet_manager.cc:1905] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.957355 752 tablet_service.cc:1468] Processing CreateTablet for tablet b90713459cf14e49b75de34942fb13f0 (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.957743 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90713459cf14e49b75de34942fb13f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.964525 3579 raft_consensus.cc:491] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.964656 3579 raft_consensus.cc:513] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.965181 3579 leader_election.cc:290] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.966572 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e3e1732bff14d49ade14fbfa6a9784c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:42.967835 848 leader_election.cc:343] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.969511 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff81146b9cd44dcba49b56ae0244f985. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:42.971554 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 56903.5us
I20250905 19:52:42.974033 3274 consensus_queue.cc:1035] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:42.974598 984 leader_election.cc:343] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:42.977066 3547 tablet_replica.cc:331] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:42.977103 3539 ts_tablet_manager.cc:1918] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:42.977272 3547 raft_consensus.cc:2241] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:42.977300 3539 log.cc:1199] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4a9afd0fd4ad4ea0afcab9d7291a76bb
I20250905 19:52:42.977451 3547 raft_consensus.cc:2270] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:42.977746 3539 ts_tablet_manager.cc:1939] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:42.978045 3547 ts_tablet_manager.cc:1905] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:42.978802 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (table default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]) successfully deleted
I20250905 19:52:42.981441 752 tablet_service.cc:1468] Processing CreateTablet for tablet 93694a3c232a48e180085cd490ee5f4d (DEFAULT_TABLE table=default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:42.981930 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93694a3c232a48e180085cd490ee5f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:42.981959 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 3 got 0
I20250905 19:52:42.985371 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:42.986163 764 raft_consensus.cc:3058] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:42.987332 766 raft_consensus.cc:1238] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Rejecting Update request from peer 4eb3af79b01d4835bf6bf7b26bf016f5 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250905 19:52:42.990051 1169 ts_tablet_manager.cc:1428] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.108s user 0.001s sys 0.001s
I20250905 19:52:42.990448 1169 tablet_bootstrap.cc:492] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:42.992157 1169 tablet_bootstrap.cc:654] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:42.993232 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:42.993875 3468 raft_consensus.cc:491] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:42.994011 3468 raft_consensus.cc:513] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:42.994585 3468 leader_election.cc:290] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:42.995098 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736bfce682e64498a8dde3310305536a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:42.996743 3678 tablet.cc:1722] T 1bd731a303a545f492b19f774520582c P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:42.998034 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:42.999198 848 leader_election.cc:343] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.001683 3557 tablet_bootstrap.cc:492] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:43.009825 3557 ts_tablet_manager.cc:1397] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250905 19:52:43.010646 3557 raft_consensus.cc:357] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.011857 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet c77dd068ea5c495ca8aa0e7a6823d792 (table default.Table_e08447fc15f5401eb931185a79ad62c8 [id=b3f6e906472a4471be3c9bdd0a25b96d]). Expected version 1 got 0
I20250905 19:52:43.013366 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:43.013748 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 1bd731a303a545f492b19f774520582c (table default.Table_7ccec4c81ed34b1b9fa8ae78c4a47b9d [id=ed7ed5bb46a84eb2bc77d1f948592244]). Expected version 1 got 0
I20250905 19:52:43.013893 3557 raft_consensus.cc:383] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.014097 3557 raft_consensus.cc:738] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.014307 3557 consensus_queue.cc:260] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.014703 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
W20250905 19:52:43.017592 849 leader_election.cc:343] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.018546 3557 ts_tablet_manager.cc:1428] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.001s sys 0.000s
I20250905 19:52:43.017971 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet d7155cf5abba4e2cbdce583406f94fad (table default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]) successfully deleted
I20250905 19:52:43.019690 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2b5ce70b784c13aa64053b2d1fe541" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:43.020399 764 raft_consensus.cc:3058] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.018169 849 leader_election.cc:304] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.021766 3557 tablet_bootstrap.cc:492] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:43.023039 3681 raft_consensus.cc:2747] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.019584 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:43.023581 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
W20250905 19:52:43.023864 849 leader_election.cc:343] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.024135 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:43.024395 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:43.024430 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2642bb409bb4473db034035b1b834739" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.024834 1099 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:43.024868 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 350197e134c549198771f4f8337d5dfc (table default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431 [id=248e84cad5c3471c947c275c89d6fe50]). Expected version 1 got 0
I20250905 19:52:43.024948 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e3cb1b098f42441594e28aeb1a9dbbf1" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.025409 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 544dffc8b77049c295921d0af091b17d (table default.Table_186206d66e7446e980e368d0716e67df [id=4044052e369843d39980a31b1654628d]). Expected version 4 got 3
I20250905 19:52:43.025544 764 raft_consensus.cc:2466] T bb2b5ce70b784c13aa64053b2d1fe541 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:43.025888 3557 tablet_bootstrap.cc:654] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.026247 849 leader_election.cc:304] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.026471 3681 raft_consensus.cc:2747] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.027490 2370 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
W20250905 19:52:43.027588 849 consensus_peers.cc:489] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 087172093fff4c41980dc2daeabb07c3. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:43.028751 3678 tablet.cc:1722] T c77dd068ea5c495ca8aa0e7a6823d792 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.029258 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e3e1732bff14d49ade14fbfa6a9784c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.029965 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736bfce682e64498a8dde3310305536a" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.036127 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6b8bbbcac06944469e13f14ab1d80402" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.036702 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.036883 1050 raft_consensus.cc:2391] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:43.037423 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a9afd0fd4ad4ea0afcab9d7291a76bb" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.037509 1170 tablet_bootstrap.cc:492] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.037631 1170 ts_tablet_manager.cc:1397] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.156s user 0.002s sys 0.008s
I20250905 19:52:43.038162 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.038261 1170 raft_consensus.cc:357] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.038427 1170 raft_consensus.cc:383] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.038479 1170 raft_consensus.cc:738] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.038676 1170 consensus_queue.cc:260] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.038748 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6edbc312fc4947a79e937dbc64d30b37" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:43.038906 1050 raft_consensus.cc:2391] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:43.043645 2342 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet e58bbb6575fd4437b242032a281914ea
I20250905 19:52:43.044068 3541 ts_tablet_manager.cc:1918] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.044644 2342 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_760e5114bbcd4203a4de6891f9510b66" } modify_external_catalogs: true
I20250905 19:52:43.044889 2342 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_760e5114bbcd4203a4de6891f9510b66" } modify_external_catalogs: true
I20250905 19:52:43.045478 1169 tablet_bootstrap.cc:492] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:43.045675 1169 ts_tablet_manager.cc:1397] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.055s user 0.000s sys 0.003s
I20250905 19:52:43.046103 2399 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet dc5428f20ca14fdf88a3a52f7b4d1e19
I20250905 19:52:43.046523 1169 raft_consensus.cc:357] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.046634 2399 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34736 (request call id 56) took 1570 ms. Trace:
I20250905 19:52:43.046880 1169 raft_consensus.cc:383] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.046993 1169 raft_consensus.cc:738] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.046758 2399 rpcz_store.cc:276] 0905 19:52:41.476555 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.996157 (+519602us) service_pool.cc:225] Handling call
0905 19:52:41.997626 (+ 1469us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:41.997793 (+ 167us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:41.998143 (+ 350us) catalog_manager.cc:2799] Locking tablets
0905 19:52:42.127477 (+129334us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:42.318496 (+191019us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:42.318609 (+ 113us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:43.046038 (+727429us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:43.046601 (+ 563us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:42.276522 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:42.276636 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:42.276642 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:42.276645 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:42.276755 (+ 110us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:42.276766 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:42.276771 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:42.276874 (+ 103us) tablet.cc:699] Row locks acquired
0905 19:52:42.276877 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:42.276912 (+ 35us) write_op.cc:270] Start()
0905 19:52:42.276951 (+ 39us) write_op.cc:276] Timestamp: P: 1757101962276715 usec, L: 0
0905 19:52:42.276954 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:42.277139 (+ 185us) log.cc:844] Serialized 759 byte log entry
0905 19:52:42.316039 (+ 38900us) op_driver.cc:464] REPLICATION: finished
0905 19:52:42.316381 (+ 342us) write_op.cc:301] APPLY: starting
0905 19:52:42.316399 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:42.316446 (+ 47us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:42.316478 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:42.316547 (+ 69us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:42.316594 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:42.316606 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:42.316710 (+ 104us) log.cc:844] Serialized 29 byte log entry
0905 19:52:42.316834 (+ 124us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.316864 (+ 30us) write_op.cc:454] Released schema lock
0905 19:52:42.316874 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:42.316968 (+ 94us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.316972 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":283,"apply.run_cpu_time_us":573,"apply.run_wall_time_us":573,"num_ops":2,"prepare.queue_time_us":148393,"prepare.run_cpu_time_us":740,"prepare.run_wall_time_us":1906,"raft.queue_time_us":37,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":5,"replication_time_us":38981}]]}
I20250905 19:52:43.047590 2399 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_084f2c1b9d144e38af5948cfd007d374" } new_table_name: "default.Table_da24a2b5f46f4c38b7cff7dcb33c3bf3" modify_external_catalogs: true
I20250905 19:52:43.047267 1169 consensus_queue.cc:260] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.046833 740 tablet_service.cc:1515] Processing DeleteTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.045424 752 tablet_service.cc:1515] Processing DeleteTablet for tablet e58bbb6575fd4437b242032a281914ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.048324 3539 tablet_replica.cc:331] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.048604 3539 raft_consensus.cc:2241] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.048810 3539 raft_consensus.cc:2270] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.049224 1050 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:43.049508 3539 ts_tablet_manager.cc:1905] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.051208 1026 tablet_service.cc:1468] Processing CreateTablet for tablet ff347675042f49ec9f31e28df7dc690d (DEFAULT_TABLE table=default.Table_5cd8ba66cf734db5afca42d7fbf46281 [id=c9ff22868d3a499e9798155a4f1448a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.051795 1019 tablet_service.cc:1468] Processing CreateTablet for tablet bd3a516f3a364c9ba30fb58b1baed332 (DEFAULT_TABLE table=default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.052172 1028 tablet_service.cc:1468] Processing CreateTablet for tablet 252afae7396d4d90b4647f4c4cc7b7cf (DEFAULT_TABLE table=default.Table_6e5d1701ba1440d7a446f7831e77d23a [id=1af821cbe6824dd7ae4ca391b805b68f]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.052369 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 29848b701d98421a81f163ffe100d8a7 (DEFAULT_TABLE table=default.Table_d58a00eae8a14ac09495cf8262d3d13c [id=1e7d0dbdca914d2ca76cc8eb85eccbc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.052464 1032 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb with delete_type TABLET_DATA_DELETED (replaced by tablet fe4cc17bdb6d4af8b323de5116aec179) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.052484 1033 tablet_service.cc:1468] Processing CreateTablet for tablet 46e5941c519a43d7ac6ba93e471e7325 (DEFAULT_TABLE table=default.Table_fe1d7268863740aea58d36b8bf9abc2c [id=706dc1f46c1c487e8911167fcc5c4a76]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.052706 1170 ts_tablet_manager.cc:1428] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.015s user 0.001s sys 0.001s
I20250905 19:52:43.052768 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 452872677922455d96126751a1b464ef (DEFAULT_TABLE table=default.Table_e2abd597f41a4e4bb4c64bcba4bebd80 [id=9cf2797afe514db487c3fbd79ddacab5]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.052979 1170 tablet_bootstrap.cc:492] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:43.053143 1031 tablet_service.cc:1468] Processing CreateTablet for tablet 6b6ae48042fe4cf99222e3b3c30a0b3f (DEFAULT_TABLE table=default.Table_8ff658df694b488b9bea5ed9bae26cf8 [id=11666097a7d8460690a76a09928574f8]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.053174 1029 tablet_service.cc:1468] Processing CreateTablet for tablet 11ed17b2bdf649a3a13b9c8657945889 (DEFAULT_TABLE table=default.Table_87c6aa6f867b47c693a3d535de50f2a2 [id=af65dfe93b874c02b16368eff59e3ef6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.053588 1023 tablet_service.cc:1468] Processing CreateTablet for tablet da588ad7d1874575a41533d298e8e2b7 (DEFAULT_TABLE table=default.Table_eae63ec4ceb6473bbd59b4a7c00e7e56 [id=088e6e4646fc4c218a4e433ce0f9fade]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.053766 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 15be6e2e31364705beb09b4a636e3782 (DEFAULT_TABLE table=default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.054046 1030 tablet_service.cc:1468] Processing CreateTablet for tablet ea1b6c96305e49b9ad025baa41ffae33 (DEFAULT_TABLE table=default.Table_942a1f8066e34bfb9e2a520e352bd6d4 [id=60f1cde785a24b4db748918ef5df5cc6]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.054538 1032 tablet_service.cc:1468] Processing CreateTablet for tablet 68e2c8572d3a4569b90664ecc4be26f2 (DEFAULT_TABLE table=default.Table_79e8c67d212c4e60be11fa071fe45c65 [id=d5ab4a8df913481b91cedb10c7171146]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.056430 1020 tablet_service.cc:1468] Processing CreateTablet for tablet ac73a74ec58b4b0b9f34af938b841aa1 (DEFAULT_TABLE table=default.Table_5597507266624e81888c14b51a6adb97 [id=1e9e241158ed4a7c9d0a808a89bf8b54]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.056926 1016 tablet_service.cc:1468] Processing CreateTablet for tablet 799c195c42184db6bab055b209e335d9 (DEFAULT_TABLE table=default.Table_c76754fbc1b94a18a9695e46eec3d7a9 [id=9878febd268641a08408e41ec155c569]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.057483 1022 tablet_service.cc:1468] Processing CreateTablet for tablet ff81146b9cd44dcba49b56ae0244f985 (DEFAULT_TABLE table=default.Table_789658eb5ab645dc90706bca2cd7eae9 [id=b87cccf4f95f4e93959679b75166ecd4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.057618 1021 tablet_service.cc:1468] Processing CreateTablet for tablet fd769c4075914d338cbee14c97402aef (DEFAULT_TABLE table=default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.058226 1017 tablet_service.cc:1468] Processing CreateTablet for tablet 73671be59bd248559fdcaf0ca2a30f16 (DEFAULT_TABLE table=default.Table_18b5523c0ad64df4b15d6f0924914671 [id=534cd54186e448f98488cc8b04e8f5ba]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.058524 1015 tablet_service.cc:1468] Processing CreateTablet for tablet 93694a3c232a48e180085cd490ee5f4d (DEFAULT_TABLE table=default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.058635 1018 tablet_service.cc:1468] Processing CreateTablet for tablet b90713459cf14e49b75de34942fb13f0 (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.059501 2397 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6fc6ab8b63df4d2684a474b096058aee" } modify_external_catalogs: true
I20250905 19:52:43.059713 2397 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6fc6ab8b63df4d2684a474b096058aee" } modify_external_catalogs: true
I20250905 19:52:43.060222 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet e58bbb6575fd4437b242032a281914ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.060483 1026 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff347675042f49ec9f31e28df7dc690d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.060542 1014 tablet_service.cc:1515] Processing DeleteTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.061836 3539 ts_tablet_manager.cc:1918] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.061933 3539 log.cc:1199] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/dc5428f20ca14fdf88a3a52f7b4d1e19
I20250905 19:52:43.062271 3539 ts_tablet_manager.cc:1939] T dc5428f20ca14fdf88a3a52f7b4d1e19 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.062810 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (table default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]) successfully deleted
I20250905 19:52:43.063700 1019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bd3a516f3a364c9ba30fb58b1baed332. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.064612 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 11ed17b2bdf649a3a13b9c8657945889. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.066097 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_a0373824c7c44e6996ef440d60dc4cca"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:43.066663 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a0373824c7c44e6996ef440d60dc4cca in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.068117 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 29848b701d98421a81f163ffe100d8a7. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.068392 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 15be6e2e31364705beb09b4a636e3782. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.068811 1030 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea1b6c96305e49b9ad025baa41ffae33. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.072062 1028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 252afae7396d4d90b4647f4c4cc7b7cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.072311 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 452872677922455d96126751a1b464ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.076126 1032 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 68e2c8572d3a4569b90664ecc4be26f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.076370 1016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 799c195c42184db6bab055b209e335d9. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.076237 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_e52d84f8329e4b4db5f19402e0963db3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:43.076524 1023 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet da588ad7d1874575a41533d298e8e2b7. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:43.077247 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e52d84f8329e4b4db5f19402e0963db3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.080199 1022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ff81146b9cd44dcba49b56ae0244f985. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.080555 1015 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93694a3c232a48e180085cd490ee5f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.080758 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ac73a74ec58b4b0b9f34af938b841aa1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.081104 1018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90713459cf14e49b75de34942fb13f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.084112 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fd769c4075914d338cbee14c97402aef. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.087738 1017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 73671be59bd248559fdcaf0ca2a30f16. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.088016 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_8acce2d2e1bf4adeaa0f35f0feabb057"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:43.088210 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6b6ae48042fe4cf99222e3b3c30a0b3f. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:43.088620 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8acce2d2e1bf4adeaa0f35f0feabb057 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.096084 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 46e5941c519a43d7ac6ba93e471e7325. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:43.097949 2372 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 51d92f92940b4952b68becf86661d459 consensus state has no leader
I20250905 19:52:43.098364 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_17b10ad086cd460faea85db26ac8dcd3"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:43.098932 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_17b10ad086cd460faea85db26ac8dcd3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.107937 3539 tablet_replica.cc:331] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.108462 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_0852970417d84646b80e1c1fd0dea533"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:43.109007 3539 raft_consensus.cc:2241] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:43.109133 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0852970417d84646b80e1c1fd0dea533 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.109190 3539 raft_consensus.cc:2270] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.109699 3539 ts_tablet_manager.cc:1905] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.117041 1170 tablet_bootstrap.cc:654] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.122228 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_9b1ba150be754450be1e96d29083d126"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:43.123327 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_9b1ba150be754450be1e96d29083d126 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.126015 3547 ts_tablet_manager.cc:1918] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:43.133020 1170 tablet_bootstrap.cc:492] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.133177 1170 ts_tablet_manager.cc:1397] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.080s user 0.000s sys 0.003s
I20250905 19:52:43.133817 1170 raft_consensus.cc:357] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.133986 1170 raft_consensus.cc:383] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.134037 1170 raft_consensus.cc:738] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.134276 1170 consensus_queue.cc:260] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.134824 2372 master_service.cc:948] ReplaceTablet: received request to replace tablet af5748bea3bd4f51a0d6c28a1d9044a0 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:43.134856 1170 ts_tablet_manager.cc:1428] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:43.135108 1170 tablet_bootstrap.cc:492] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:43.136155 3539 ts_tablet_manager.cc:1918] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:43.136334 3539 log.cc:1199] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e58bbb6575fd4437b242032a281914ea
I20250905 19:52:43.136757 3539 ts_tablet_manager.cc:1939] T e58bbb6575fd4437b242032a281914ea P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.136921 2340 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet c69a41cee5ff4f8f9bccaa513c096118
I20250905 19:52:43.137583 2340 master_service.cc:948] ReplaceTablet: received request to replace tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:43.136837 1170 tablet_bootstrap.cc:654] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.130069 3151 consensus_queue.cc:1035] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.131196 1169 ts_tablet_manager.cc:1428] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.085s user 0.000s sys 0.001s
I20250905 19:52:43.138914 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]) successfully deleted
I20250905 19:52:43.139231 1169 tablet_bootstrap.cc:492] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:43.139813 3547 log.cc:1199] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/ad0760dfee934aacbd4ab1fb81e116c7
I20250905 19:52:43.139786 752 tablet_service.cc:1515] Processing DeleteTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.140071 3539 tablet_replica.cc:331] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.140126 2403 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_cae023e67f764978833e906977ad0dde [id=ec8ef6059b414e5189d80735b0cde61a] alter complete (version 1)
I20250905 19:52:43.140321 3539 raft_consensus.cc:2241] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.140489 3539 raft_consensus.cc:2270] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.141008 3539 ts_tablet_manager.cc:1905] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.143447 3547 ts_tablet_manager.cc:1939] T ad0760dfee934aacbd4ab1fb81e116c7 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:43.144537 3547 tablet_replica.cc:331] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:43.144758 3547 raft_consensus.cc:2241] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.144942 3547 raft_consensus.cc:2270] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.145563 3547 ts_tablet_manager.cc:1905] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.147338 3539 ts_tablet_manager.cc:1918] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.147452 3539 log.cc:1199] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/c69a41cee5ff4f8f9bccaa513c096118
I20250905 19:52:43.147816 3539 ts_tablet_manager.cc:1939] T c69a41cee5ff4f8f9bccaa513c096118 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.141376 3541 log.cc:1199] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d1a11777cf2d4ea9a5cdf792f73aff76
I20250905 19:52:43.150394 3541 ts_tablet_manager.cc:1939] T d1a11777cf2d4ea9a5cdf792f73aff76 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:43.151091 2405 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b], current schema version=5): Not found: failed to reset TS proxy: No leader found: Tablet 9ca3fc7055c44df282285806ea151e51 consensus state has no leader
W20250905 19:52:43.151638 2405 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet bd9ae8fd5fdc4b539308268ba9729cd6 consensus state has no leader
I20250905 19:52:43.153517 1170 tablet_bootstrap.cc:492] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.153648 1170 ts_tablet_manager.cc:1397] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.019s user 0.000s sys 0.003s
I20250905 19:52:43.154259 1170 raft_consensus.cc:357] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.154431 1170 raft_consensus.cc:383] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.154474 1170 raft_consensus.cc:738] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.154680 1170 consensus_queue.cc:260] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.155519 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b with delete_type TABLET_DATA_DELETED (replaced by tablet b23111dc8af94b56af9538f973248f2a) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.155725 744 tablet_service.cc:1515] Processing DeleteTablet for tablet fb72fe6049894527917e1481bcb7b03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:38 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.155886 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.156054 740 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.156220 743 tablet_service.cc:1515] Processing DeleteTablet for tablet e67ed7e9e5724de389504701584bd2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.156366 745 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 with delete_type TABLET_DATA_DELETED (replaced by tablet bd3a516f3a364c9ba30fb58b1baed332) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.156054 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 with delete_type TABLET_DATA_DELETED (replaced by tablet 26905425077e4746bccb51431da706c9) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.156594 744 tablet_service.cc:1515] Processing DeleteTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 with delete_type TABLET_DATA_DELETED (replaced by tablet 98db7e2a9d6d4f4d984388a493295d72) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.156965 753 tablet_service.cc:1515] Processing DeleteTablet for tablet 087172093fff4c41980dc2daeabb07c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.157124 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.157595 1170 ts_tablet_manager.cc:1428] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:43.157966 1170 tablet_bootstrap.cc:492] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:43.159298 2405 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:45391 (request call id 132) took 1659 ms (client timeout 1000 ms). Trace:
I20250905 19:52:43.159919 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d with delete_type TABLET_DATA_DELETED (replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4) from {username='slave'} at 127.0.0.1:37950
W20250905 19:52:43.159492 2405 rpcz_store.cc:269] 0905 19:52:41.500137 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.506725 (+ 6588us) service_pool.cc:225] Handling call
0905 19:52:41.532782 (+ 26057us) spinlock_profiling.cc:243] Waited 21.3 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1bf4a7d 00007f73d1d786f8 00007f73c8bb2c86 00007f73c8bb29b5 00007f73c538d0be 00007f73c538bf9e 00007f73c53921d5 00007f73c5392aff
0905 19:52:41.804814 (+272032us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.804822 (+ 8us) write_op.cc:454] Released schema lock
0905 19:52:43.159251 (+1354429us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.796362 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.796497 (+ 135us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.796504 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:41.796506 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:41.796701 (+ 195us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.796715 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.796720 (+ 5us) tablet.cc:683] Acquiring locks for 5 operations
0905 19:52:41.796875 (+ 155us) tablet.cc:699] Row locks acquired
0905 19:52:41.796877 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:41.796916 (+ 39us) write_op.cc:270] Start()
0905 19:52:41.796951 (+ 35us) write_op.cc:276] Timestamp: P: 1757101961796839 usec, L: 0
0905 19:52:41.796955 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.797146 (+ 191us) log.cc:844] Serialized 2024 byte log entry
0905 19:52:41.801610 (+ 4464us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.801744 (+ 134us) write_op.cc:301] APPLY: starting
0905 19:52:41.801759 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.801856 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.801879 (+ 23us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.801992 (+ 113us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.802029 (+ 37us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=5
0905 19:52:41.802037 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:41.802129 (+ 92us) log.cc:844] Serialized 47 byte log entry
0905 19:52:41.802191 (+ 62us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.802217 (+ 26us) write_op.cc:454] Released schema lock
0905 19:52:41.802224 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:41.939501 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.939594 (+ 93us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.939598 (+ 4us) write_op.cc:435] Acquired schema lock
0905 19:52:41.939600 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:41.939678 (+ 78us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.939688 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.939693 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:41.939741 (+ 48us) tablet.cc:699] Row locks acquired
0905 19:52:41.939743 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:41.939774 (+ 31us) write_op.cc:270] Start()
0905 19:52:41.939800 (+ 26us) write_op.cc:276] Timestamp: P: 1757101961939698 usec, L: 0
0905 19:52:41.939803 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.939958 (+ 155us) log.cc:844] Serialized 356 byte log entry
0905 19:52:41.952915 (+ 12957us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.953530 (+ 615us) write_op.cc:301] APPLY: starting
0905 19:52:41.953537 (+ 7us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.953562 (+ 25us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.953590 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.953620 (+ 30us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.953641 (+ 21us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:41.953648 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:41.953706 (+ 58us) log.cc:844] Serialized 23 byte log entry
0905 19:52:41.953735 (+ 29us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.953746 (+ 11us) write_op.cc:454] Released schema lock
0905 19:52:41.953751 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.984507 (+ 30756us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.984516 (+ 9us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:42.200123 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:42.200243 (+ 120us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:42.200249 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:42.200251 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:42.200352 (+ 101us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:42.200363 (+ 11us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:42.200369 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:42.200469 (+ 100us) tablet.cc:699] Row locks acquired
0905 19:52:42.200471 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:42.200508 (+ 37us) write_op.cc:270] Start()
0905 19:52:42.200541 (+ 33us) write_op.cc:276] Timestamp: P: 1757101962200310 usec, L: 0
0905 19:52:42.200544 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:42.200724 (+ 180us) log.cc:844] Serialized 356 byte log entry
0905 19:52:42.226591 (+ 25867us) op_driver.cc:464] REPLICATION: finished
0905 19:52:42.227100 (+ 509us) write_op.cc:301] APPLY: starting
0905 19:52:42.227134 (+ 34us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:42.227173 (+ 39us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:42.227207 (+ 34us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:42.227258 (+ 51us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:42.227288 (+ 30us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:42.227296 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:42.227364 (+ 68us) log.cc:844] Serialized 23 byte log entry
0905 19:52:42.227421 (+ 57us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.227439 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:42.227446 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:42.227516 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.227520 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":57,"reported_tablets":159,"spinlock_wait_cycles":49077120,"child_traces":[["op",{"apply.queue_time_us":86,"apply.run_cpu_time_us":535,"apply.run_wall_time_us":534,"num_ops":5,"prepare.queue_time_us":51,"prepare.run_cpu_time_us":887,"prepare.run_wall_time_us":888,"raft.queue_time_us":1372,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":251,"replication_time_us":4621}],["op",{"apply.queue_time_us":589,"apply.run_cpu_time_us":262,"apply.run_wall_time_us":262,"num_ops":1,"prepare.queue_time_us":999,"prepare.run_cpu_time_us":2927,"prepare.run_wall_time_us":44983,"raft.queue_time_us":933,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":13099,"thread_start_us":43965,"threads_started":1}],["op",{"apply.queue_time_us":462,"apply.run_cpu_time_us":406,"apply.run_wall_time_us":406,"num_ops":1,"prepare.queue_time_us":81457,"prepare.run_cpu_time_us":758,"prepare.run_wall_time_us":8203,"raft.queue_time_us":170,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":26013}]]}
I20250905 19:52:43.162065 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.162546 744 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.162565 1170 tablet_bootstrap.cc:654] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.163391 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.163499 753 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf448f9abc348df8d2d2501420d8f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.163636 745 tablet_service.cc:1515] Processing DeleteTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f with delete_type TABLET_DATA_DELETED (replaced by tablet fe00969eb9654976a0efeac2fa7c91d3) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.163651 986 leader_election.cc:304] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.163740 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 with delete_type TABLET_DATA_DELETED (replaced by tablet 350197e134c549198771f4f8337d5dfc) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.164022 752 tablet_service.cc:1515] Processing DeleteTablet for tablet be1315824fd749049a62af3e6a113b9a with delete_type TABLET_DATA_DELETED (replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.164316 3151 raft_consensus.cc:2802] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:43.164405 3539 tablet_replica.cc:331] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.164489 3151 raft_consensus.cc:695] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:43.164661 3539 raft_consensus.cc:2241] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.163184 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet c69a41cee5ff4f8f9bccaa513c096118 (table default.Table_a98611b77e2d4e7c868fe9641069ae23 [id=b721335ab8dd437a9ecb855b872ad09d]) successfully deleted
I20250905 19:52:43.164880 3151 consensus_queue.cc:237] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
W20250905 19:52:43.165086 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 5a59d7fb5dc54b479d965282b1c7d24b because the tablet was not found. No further retry: Not found: Tablet not found: 5a59d7fb5dc54b479d965282b1c7d24b
I20250905 19:52:43.165832 3539 raft_consensus.cc:2270] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.166415 3539 ts_tablet_manager.cc:1905] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.166393 2405 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_d7da0965c1624d51adff3f0ef965be87"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:43.167045 2405 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_d7da0965c1624d51adff3f0ef965be87 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.127012 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "51d92f92940b4952b68becf86661d459" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:43.167295 907 raft_consensus.cc:3058] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:43.170042 907 raft_consensus.cc:2466] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 2.
I20250905 19:52:43.128409 877 tablet_service.cc:1515] Processing DeleteTablet for tablet e58bbb6575fd4437b242032a281914ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.128782 882 tablet_service.cc:1468] Processing CreateTablet for tablet b90713459cf14e49b75de34942fb13f0 (DEFAULT_TABLE table=default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.170647 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b90713459cf14e49b75de34942fb13f0. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.129127 875 tablet_service.cc:1468] Processing CreateTablet for tablet 93694a3c232a48e180085cd490ee5f4d (DEFAULT_TABLE table=default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.172251 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93694a3c232a48e180085cd490ee5f4d. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.172611 3666 raft_consensus.cc:491] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.172783 3666 raft_consensus.cc:513] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.173518 3666 leader_election.cc:290] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.131762 883 tablet_service.cc:1515] Processing DeleteTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
W20250905 19:52:43.124893 848 consensus_peers.cc:489] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad0760dfee934aacbd4ab1fb81e116c7. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:43.177274 848 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 52423.8us
W20250905 19:52:43.167091 849 leader_election.cc:343] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.183825 849 leader_election.cc:304] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.184367 3468 raft_consensus.cc:2747] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:43.184908 849 leader_election.cc:343] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.185022 849 leader_election.cc:304] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.185217 3468 raft_consensus.cc:2747] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.187255 1170 tablet_bootstrap.cc:492] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.187395 1170 ts_tablet_manager.cc:1397] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.030s user 0.003s sys 0.003s
I20250905 19:52:43.188649 712 leader_election.cc:304] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:43.189074 3666 raft_consensus.cc:2802] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 FOLLOWER]: Leader election won for term 2
I20250905 19:52:43.189196 3666 raft_consensus.cc:695] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [term 2 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:43.189200 1170 raft_consensus.cc:357] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.189543 1170 raft_consensus.cc:383] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.189467 3666 consensus_queue.cc:237] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.189589 1170 raft_consensus.cc:738] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.189659 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.189766 1170 consensus_queue.cc:260] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.189824 905 raft_consensus.cc:2466] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:43.190140 1170 ts_tablet_manager.cc:1428] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20250905 19:52:43.190400 1170 tablet_bootstrap.cc:492] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:43.196715 848 consensus_peers.cc:489] T fb72fe6049894527917e1481bcb7b03a P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:43.201319 1170 tablet_bootstrap.cc:654] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.202231 882 tablet_service.cc:1515] Processing DeleteTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.204136 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_3244f6924a9e487d8f33dd11032916ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:43.219723 3641 raft_consensus.cc:491] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.219892 3641 raft_consensus.cc:513] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.220557 3641 leader_election.cc:290] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.220912 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a104b1c8ce14fa2bf101e1b34183cfa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:43.221683 848 leader_election.cc:343] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.221992 3641 raft_consensus.cc:491] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.222106 3641 raft_consensus.cc:513] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.222584 3641 leader_election.cc:290] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.223140 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ca3fc7055c44df282285806ea151e51" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:43.223619 848 leader_election.cc:343] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:43.225252 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3244f6924a9e487d8f33dd11032916ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.225435 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 16) took 1585 ms. Trace:
I20250905 19:52:43.225534 3541 rpcz_store.cc:276] 0905 19:52:41.640015 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.640300 (+ 285us) service_pool.cc:225] Handling call
0905 19:52:42.914046 (+1273746us) ts_tablet_manager.cc:1145] Deleting tablet d1a11777cf2d4ea9a5cdf792f73aff76
0905 19:52:42.914055 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:42.959849 (+ 45794us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:43.043937 (+ 84088us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:43.225411 (+181474us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":45114,"tablet-delete.queue_time_us":1203902}
I20250905 19:52:43.227887 711 leader_election.cc:304] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:43.228219 3546 raft_consensus.cc:2747] T 2642bb409bb4473db034035b1b834739 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.228677 711 leader_election.cc:304] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:43.228905 3546 raft_consensus.cc:2747] T e3cb1b098f42441594e28aeb1a9dbbf1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.229300 711 leader_election.cc:304] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:43.229547 3546 raft_consensus.cc:3058] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.232594 3546 raft_consensus.cc:2747] T 6b8bbbcac06944469e13f14ab1d80402 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.233114 711 leader_election.cc:304] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:43.233268 711 raft_consensus.cc:2684] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
W20250905 19:52:43.233834 711 leader_election.cc:343] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 3c824f5591044c4198a51a25661f71e5
I20250905 19:52:43.233928 711 leader_election.cc:304] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:43.234114 3546 raft_consensus.cc:2747] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20250905 19:52:43.234791 711 leader_election.cc:304] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:43.235227 3546 raft_consensus.cc:2747] T 6edbc312fc4947a79e937dbc64d30b37 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.237102 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.237282 905 raft_consensus.cc:2466] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:43.237700 905 raft_consensus.cc:1238] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Rejecting Update request from peer 4eb3af79b01d4835bf6bf7b26bf016f5 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20250905 19:52:43.237954 3546 raft_consensus.cc:491] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.238093 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "bb2b5ce70b784c13aa64053b2d1fe541" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:43.238102 3546 raft_consensus.cc:513] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.238224 905 raft_consensus.cc:3058] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.238925 3546 leader_election.cc:290] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.241044 905 raft_consensus.cc:2466] T bb2b5ce70b784c13aa64053b2d1fe541 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:43.142932 1169 tablet_bootstrap.cc:654] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.245034 3299 raft_consensus.cc:491] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.245230 3299 raft_consensus.cc:513] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.245847 3299 leader_election.cc:290] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.246037 3539 ts_tablet_manager.cc:1918] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.249282 3539 log.cc:1199] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/fb72fe6049894527917e1481bcb7b03a
I20250905 19:52:43.249961 3539 ts_tablet_manager.cc:1939] T fb72fe6049894527917e1481bcb7b03a P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.254487 3536 consensus_queue.cc:1046] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.254822 3274 raft_consensus.cc:3053] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Stepping down as leader of term 1
I20250905 19:52:43.254909 3274 raft_consensus.cc:738] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:43.255165 3274 consensus_queue.cc:260] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.255789 3607 tablet.cc:1722] T 6597c91bec4f4143ab3ed40afe53a35e P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.257313 3539 tablet_replica.cc:331] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.257367 3677 tablet.cc:1722] T 9f953a7490434bc7afa5510c33e79401 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.257831 3539 raft_consensus.cc:2241] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.258222 3539 raft_consensus.cc:2270] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.258878 3539 ts_tablet_manager.cc:1905] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.261420 3274 raft_consensus.cc:3058] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Advancing to term 2
I20250905 19:52:43.265991 3274 consensus_queue.cc:1035] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.231614 2397 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 0039b8f0ba704907b2820c9e3a69cd73
I20250905 19:52:43.269897 2397 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34736 (request call id 81) took 1017 ms. Trace:
I20250905 19:52:43.269999 2397 rpcz_store.cc:276] 0905 19:52:42.252621 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.059326 (+806705us) service_pool.cc:225] Handling call
0905 19:52:43.059851 (+ 525us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:43.060023 (+ 172us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:43.060082 (+ 59us) catalog_manager.cc:2799] Locking tablets
0905 19:52:43.205496 (+145414us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:43.231366 (+ 25870us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:43.231470 (+ 104us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:43.231551 (+ 81us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:43.269844 (+ 38293us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.206480 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.206648 (+ 168us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.206658 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:43.206662 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:43.206834 (+ 172us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.206853 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.206862 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:43.207012 (+ 150us) tablet.cc:699] Row locks acquired
0905 19:52:43.207016 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:43.207063 (+ 47us) write_op.cc:270] Start()
0905 19:52:43.207109 (+ 46us) write_op.cc:276] Timestamp: P: 1757101963207042 usec, L: 0
0905 19:52:43.207115 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.207352 (+ 237us) log.cc:844] Serialized 759 byte log entry
0905 19:52:43.230457 (+ 23105us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.230636 (+ 179us) write_op.cc:301] APPLY: starting
0905 19:52:43.230660 (+ 24us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.230737 (+ 77us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.230782 (+ 45us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.230878 (+ 96us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.230927 (+ 49us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:43.230938 (+ 11us) write_op.cc:312] APPLY: finished
0905 19:52:43.231036 (+ 98us) log.cc:844] Serialized 29 byte log entry
0905 19:52:43.231229 (+ 193us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.231257 (+ 28us) write_op.cc:454] Released schema lock
0905 19:52:43.231268 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:43.240597 (+ 9329us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.240605 (+ 8us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":116,"apply.run_cpu_time_us":628,"apply.run_wall_time_us":9942,"num_ops":2,"prepare.queue_time_us":67,"prepare.run_cpu_time_us":1048,"prepare.run_wall_time_us":1906,"raft.queue_time_us":67,"raft.run_cpu_time_us":298,"raft.run_wall_time_us":301,"replication_time_us":23298}]]}
I20250905 19:52:43.270789 2397 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_3dafac26db484c73a9482d5732d1fbf4"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:43.271342 2397 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_3dafac26db484c73a9482d5732d1fbf4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.272190 3541 tablet_replica.cc:331] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:43.272414 3541 raft_consensus.cc:2241] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.272584 3541 raft_consensus.cc:2270] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.273204 3541 ts_tablet_manager.cc:1905] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:43.235379 2403 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:55401 (request call id 77) took 2124 ms (client timeout 1000 ms). Trace:
I20250905 19:52:43.275391 3539 ts_tablet_manager.cc:1918] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20250905 19:52:43.273989 2403 rpcz_store.cc:269] 0905 19:52:41.110733 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.130539 (+ 19806us) service_pool.cc:225] Handling call
0905 19:52:41.139806 (+ 9267us) spinlock_profiling.cc:243] Waited 5.44 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1bf4a7d 00007f73d1d786f8 00007f73c8bb2c86 00007f73c8bb29b5 00007f73c538d0be 00007f73c538bf9e 00007f73c53921d5 00007f73c5392aff
0905 19:52:41.691257 (+551451us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.691266 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:43.235336 (+1544070us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:41.680759 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.680910 (+ 151us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.680920 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:41.680924 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:41.681546 (+ 622us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.681566 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.681574 (+ 8us) tablet.cc:683] Acquiring locks for 22 operations
0905 19:52:41.682201 (+ 627us) tablet.cc:699] Row locks acquired
0905 19:52:41.682206 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:41.682255 (+ 49us) write_op.cc:270] Start()
0905 19:52:41.682300 (+ 45us) write_op.cc:276] Timestamp: P: 1757101961682174 usec, L: 0
0905 19:52:41.682306 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.682567 (+ 261us) log.cc:844] Serialized 8382 byte log entry
0905 19:52:41.686464 (+ 3897us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.686625 (+ 161us) write_op.cc:301] APPLY: starting
0905 19:52:41.686644 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.686772 (+ 128us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.686813 (+ 41us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.687277 (+ 464us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.687344 (+ 67us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=22
0905 19:52:41.687355 (+ 11us) write_op.cc:312] APPLY: finished
0905 19:52:41.687503 (+ 148us) log.cc:844] Serialized 152 byte log entry
0905 19:52:41.687574 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.687627 (+ 53us) write_op.cc:454] Released schema lock
0905 19:52:41.687638 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:41.938785 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:41.938899 (+ 114us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:41.938905 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:41.938907 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:41.939016 (+ 109us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:41.939030 (+ 14us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:41.939035 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:41.939100 (+ 65us) tablet.cc:699] Row locks acquired
0905 19:52:41.939103 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:41.939150 (+ 47us) write_op.cc:270] Start()
0905 19:52:41.939183 (+ 33us) write_op.cc:276] Timestamp: P: 1757101961939072 usec, L: 0
0905 19:52:41.939187 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:41.939364 (+ 177us) log.cc:844] Serialized 356 byte log entry
0905 19:52:41.952852 (+ 13488us) op_driver.cc:464] REPLICATION: finished
0905 19:52:41.953071 (+ 219us) write_op.cc:301] APPLY: starting
0905 19:52:41.953088 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:41.953138 (+ 50us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:41.953163 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:41.953217 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:41.953248 (+ 31us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:41.953256 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:41.953331 (+ 75us) log.cc:844] Serialized 23 byte log entry
0905 19:52:41.953387 (+ 56us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.953405 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:41.953411 (+ 6us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:41.953470 (+ 59us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:41.953473 (+ 3us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:42.072696 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:42.072825 (+ 129us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:42.072830 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:42.072832 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:42.072929 (+ 97us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:42.072942 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:42.072947 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:42.073009 (+ 62us) tablet.cc:699] Row locks acquired
0905 19:52:42.073012 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:42.073044 (+ 32us) write_op.cc:270] Start()
0905 19:52:42.073075 (+ 31us) write_op.cc:276] Timestamp: P: 1757101962072848 usec, L: 0
0905 19:52:42.073079 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:42.073263 (+ 184us) log.cc:844] Serialized 356 byte log entry
0905 19:52:42.110810 (+ 37547us) op_driver.cc:464] REPLICATION: finished
0905 19:52:42.111335 (+ 525us) write_op.cc:301] APPLY: starting
0905 19:52:42.111354 (+ 19us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:42.111399 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:42.111425 (+ 26us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:42.111496 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:42.111538 (+ 42us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:42.111550 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:42.111646 (+ 96us) log.cc:844] Serialized 23 byte log entry
0905 19:52:42.111723 (+ 77us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.111747 (+ 24us) write_op.cc:454] Released schema lock
0905 19:52:42.111757 (+ 10us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:42.111842 (+ 85us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.111848 (+ 6us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:42.208346 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:42.209215 (+ 869us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:42.209220 (+ 5us) write_op.cc:435] Acquired schema lock
0905 19:52:42.209222 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:42.210143 (+ 921us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:42.210153 (+ 10us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:42.210158 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:42.210215 (+ 57us) tablet.cc:699] Row locks acquired
0905 19:52:42.210218 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:42.210255 (+ 37us) write_op.cc:270] Start()
0905 19:52:42.210287 (+ 32us) write_op.cc:276] Timestamp: P: 1757101962210058 usec, L: 0
0905 19:52:42.210291 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:42.210474 (+ 183us) log.cc:844] Serialized 356 byte log entry
0905 19:52:42.226924 (+ 16450us) op_driver.cc:464] REPLICATION: finished
0905 19:52:42.227583 (+ 659us) write_op.cc:301] APPLY: starting
0905 19:52:42.227593 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:42.227623 (+ 30us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:42.227656 (+ 33us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:42.227692 (+ 36us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:42.227716 (+ 24us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:42.227723 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:42.227793 (+ 70us) log.cc:844] Serialized 23 byte log entry
0905 19:52:42.227830 (+ 37us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.227844 (+ 14us) write_op.cc:454] Released schema lock
0905 19:52:42.227849 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:42.244576 (+ 16727us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.244587 (+ 11us) write_op.cc:454] Released schema lock
Related trace 'op':
0905 19:52:42.333909 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:42.334047 (+ 138us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:42.334053 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:42.334055 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:42.334193 (+ 138us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:42.334206 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:42.334212 (+ 6us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:42.334280 (+ 68us) tablet.cc:699] Row locks acquired
0905 19:52:42.334282 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:42.334319 (+ 37us) write_op.cc:270] Start()
0905 19:52:42.334356 (+ 37us) write_op.cc:276] Timestamp: P: 1757101962334121 usec, L: 0
0905 19:52:42.334360 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:42.435665 (+101305us) log.cc:844] Serialized 356 byte log entry
0905 19:52:42.441399 (+ 5734us) spinlock_profiling.cc:243] Waited 101 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f73ca03bd62 00007f73cb3e42d5 00007f73cb3e2df9 00007f73cb3eb220 00007f73cb3eb02d 0000000000a774dc 00007f73be5ea14f 00007f73be5ed41b 00007f73be5ed298 0000000000a774dc
0905 19:52:42.477913 (+ 36514us) op_driver.cc:464] REPLICATION: finished
0905 19:52:42.478633 (+ 720us) write_op.cc:301] APPLY: starting
0905 19:52:42.478645 (+ 12us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:42.478682 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:42.478684 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:42.478744 (+ 60us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:42.478770 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:42.478777 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:42.478876 (+ 99us) log.cc:844] Serialized 23 byte log entry
0905 19:52:42.478924 (+ 48us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.478937 (+ 13us) write_op.cc:454] Released schema lock
0905 19:52:42.478943 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":141,"spinlock_wait_cycles":12513152,"child_traces":[["op",{"apply.queue_time_us":103,"apply.run_cpu_time_us":1078,"apply.run_wall_time_us":1077,"num_ops":22,"prepare.queue_time_us":54,"prepare.run_cpu_time_us":1955,"prepare.run_wall_time_us":1962,"raft.queue_time_us":1231,"raft.run_cpu_time_us":334,"raft.run_wall_time_us":333,"replication_time_us":4119}],["op",{"apply.queue_time_us":175,"apply.run_cpu_time_us":390,"apply.run_wall_time_us":391,"num_ops":1,"prepare.queue_time_us":3304,"prepare.run_cpu_time_us":703,"prepare.run_wall_time_us":703,"raft.queue_time_us":1557,"raft.run_cpu_time_us":230,"raft.run_wall_time_us":229,"replication_time_us":13637}],["op",{"apply.queue_time_us":465,"apply.run_cpu_time_us":491,"apply.run_wall_time_us":491,"num_ops":1,"prepare.queue_time_us":26213,"prepare.run_cpu_time_us":768,"prepare.run_wall_time_us":10091,"raft.queue_time_us":1989,"raft.run_cpu_time_us":265,"raft.run_wall_time_us":265,"replication_time_us":37680}],["op",{"apply.queue_time_us":626,"apply.run_cpu_time_us":318,"apply.run_wall_time_us":318,"num_ops":1,"prepare.queue_time_us":84418,"prepare.run_cpu_time_us":812,"prepare.run_wall_time_us":36158,"raft.queue_time_us":231,"raft.run_cpu_time_us":122,"raft.run_wall_time_us":122,"replication_time_us":16608}],["op",{"apply.queue_time_us":677,"apply.run_cpu_time_us":367,"apply.run_wall_time_us":367,"num_ops":1,"prepare.queue_time_us":99363,"prepare.run_cpu_time_us":1830,"prepare.run_wall_time_us":117573,"raft.queue_time_us":68,"raft.run_cpu_time_us":266,"raft.run_wall_time_us":268,"replication_time_us":143526,"spinlock_wait_cycles":232533248}]]}
I20250905 19:52:43.279215 3274 consensus_queue.cc:1035] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.281529 960 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
W20250905 19:52:43.289938 984 leader_election.cc:343] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.292590 3539 log.cc:1199] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6ebab2e3698f4f60a770bde9cc019600
I20250905 19:52:43.293078 3539 ts_tablet_manager.cc:1939] T 6ebab2e3698f4f60a770bde9cc019600 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.294266 3539 tablet_replica.cc:331] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.294750 3539 raft_consensus.cc:2241] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
W20250905 19:52:43.239467 2373 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:45433 (request call id 120) took 1612 ms (client timeout 1000 ms). Trace:
I20250905 19:52:43.249243 2340 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 3742e3b46ed24b9082fa3ba5a8e5ee29
I20250905 19:52:43.295678 3539 raft_consensus.cc:2270] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.295742 2340 catalog_manager.cc:6652] ReplaceTablet: tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 deleted and replaced by tablet f1071206a7cd418f81bf2ce4eaab5237
I20250905 19:52:43.296129 2340 master_service.cc:948] ReplaceTablet: received request to replace tablet c69a41cee5ff4f8f9bccaa513c096118 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:43.296352 3539 ts_tablet_manager.cc:1905] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.296627 2340 master_service.cc:948] ReplaceTablet: received request to replace tablet 607c8a8848304491bbd4a7971f96322f from {username='slave'} at 127.0.0.1:34736
W20250905 19:52:43.298197 2373 rpcz_store.cc:269] 0905 19:52:41.626738 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.630201 (+ 3463us) service_pool.cc:225] Handling call
0905 19:52:43.239416 (+1609215us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:42.082857 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:42.083026 (+ 169us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:42.083036 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:42.083040 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:42.083444 (+ 404us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:42.083461 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:42.083470 (+ 9us) tablet.cc:683] Acquiring locks for 12 operations
0905 19:52:42.083869 (+ 399us) tablet.cc:699] Row locks acquired
0905 19:52:42.083873 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:42.083921 (+ 48us) write_op.cc:270] Start()
0905 19:52:42.083984 (+ 63us) write_op.cc:276] Timestamp: P: 1757101962083720 usec, L: 0
0905 19:52:42.083990 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:42.084248 (+ 258us) log.cc:844] Serialized 4642 byte log entry
0905 19:52:42.111236 (+ 26988us) op_driver.cc:464] REPLICATION: finished
0905 19:52:42.111929 (+ 693us) write_op.cc:301] APPLY: starting
0905 19:52:42.111939 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:42.112036 (+ 97us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:42.112076 (+ 40us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:42.112230 (+ 154us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:42.112263 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=12
0905 19:52:42.112269 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:42.112353 (+ 84us) log.cc:844] Serialized 89 byte log entry
0905 19:52:42.112398 (+ 45us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.112429 (+ 31us) write_op.cc:454] Released schema lock
0905 19:52:42.112440 (+ 11us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:42.131050 (+ 18610us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:42.131061 (+ 11us) write_op.cc:454] Released schema lock
Metrics: {"reported_tablets":142,"child_traces":[["op",{"apply.queue_time_us":647,"apply.run_cpu_time_us":581,"apply.run_wall_time_us":580,"num_ops":12,"prepare.queue_time_us":29487,"prepare.run_cpu_time_us":1606,"prepare.run_wall_time_us":48192,"raft.queue_time_us":169,"raft.run_cpu_time_us":110,"raft.run_wall_time_us":110,"replication_time_us":27217}]]}
I20250905 19:52:43.299312 2373 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_ce82d3ecb4894b59aa7e82b457011d4a" } new_table_name: "default.Table_6bb624ff162648cda8ec7badc63433ed" modify_external_catalogs: true
I20250905 19:52:43.299557 984 leader_election.cc:304] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:43.299815 3299 raft_consensus.cc:2802] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:43.300017 3299 raft_consensus.cc:695] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:43.300310 3299 consensus_queue.cc:237] T bb2b5ce70b784c13aa64053b2d1fe541 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.302976 3678 tablet.cc:1722] T 6597c91bec4f4143ab3ed40afe53a35e P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.305189 3666 consensus_queue.cc:1035] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.306425 2403 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_4fe8c2a1caf340069ff229767f87e772" } new_table_name: "default.Table_234e811e09aa42e2a83b43d09c7d8ee0" modify_external_catalogs: true
I20250905 19:52:43.307709 3678 tablet.cc:1722] T 9f953a7490434bc7afa5510c33e79401 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250905 19:52:43.255121 2285 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet bd9ae8fd5fdc4b539308268ba9729cd6 consensus state has no leader
I20250905 19:52:43.309829 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 with delete_type TABLET_DATA_DELETED (replaced by tablet f1071206a7cd418f81bf2ce4eaab5237) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.310338 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet fb72fe6049894527917e1481bcb7b03a (table default.Table_191f4f9b06184a8a8429cce5802c520e [id=a9604dfc1b44424f97da39857b13ae91]) successfully deleted
I20250905 19:52:43.310616 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6ebab2e3698f4f60a770bde9cc019600 (table default.Table_68450db54c5641a1a249cdcf5fa2d5ec [id=94f2531a27494c32bcd549a25287867f]) successfully deleted
I20250905 19:52:43.314635 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.315095 905 raft_consensus.cc:2466] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:43.319399 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 0039b8f0ba704907b2820c9e3a69cd73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.323377 3539 ts_tablet_manager.cc:1918] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.327100 3536 consensus_queue.cc:1035] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.328354 986 leader_election.cc:304] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.328891 3601 raft_consensus.cc:2802] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:43.329062 3601 raft_consensus.cc:491] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:43.329210 3601 raft_consensus.cc:3058] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.340121 3539 log.cc:1199] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/3c824f5591044c4198a51a25661f71e5
I20250905 19:52:43.340763 3539 ts_tablet_manager.cc:1939] T 3c824f5591044c4198a51a25661f71e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.341640 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 3c824f5591044c4198a51a25661f71e5 (table default.Table_4ec178bc65804a6fb36bb46ecfe8378f [id=98189a25fdcf451e89d8800fedae7d62]) successfully deleted
W20250905 19:52:43.342911 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e67ed7e9e5724de389504701584bd2b6 because the tablet was not found. No further retry: Not found: Tablet not found: e67ed7e9e5724de389504701584bd2b6
W20250905 19:52:43.343839 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet d1a11777cf2d4ea9a5cdf792f73aff76 because the tablet was not found. No further retry: Not found: Tablet not found: d1a11777cf2d4ea9a5cdf792f73aff76
W20250905 19:52:43.344537 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9a81376d4c0b47a9817deae4d34ffb47 because the tablet was not found. No further retry: Not found: Tablet not found: 9a81376d4c0b47a9817deae4d34ffb47
I20250905 19:52:43.344875 3539 tablet_replica.cc:331] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.345145 3539 raft_consensus.cc:2241] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.345441 3539 raft_consensus.cc:2270] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.346169 3539 ts_tablet_manager.cc:1905] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.349850 3601 raft_consensus.cc:513] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.356513 3539 ts_tablet_manager.cc:1918] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:43.356760 3539 log.cc:1199] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:43.357158 3539 ts_tablet_manager.cc:1939] T e6ac13b19fb6427a9ef35a5160033a54 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.359629 3151 consensus_queue.cc:1035] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.360752 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet e6ac13b19fb6427a9ef35a5160033a54 (table default.Table_515c33e99f2c486abcf9e9333253193b [id=becebe5644f44086aac02f9ea9281e5f]) successfully deleted
W20250905 19:52:43.361637 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 087172093fff4c41980dc2daeabb07c3 because the tablet was not found. No further retry: Not found: Tablet not found: 087172093fff4c41980dc2daeabb07c3
I20250905 19:52:43.361928 3539 tablet_replica.cc:331] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.362174 3539 raft_consensus.cc:2241] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.362375 3539 raft_consensus.cc:2270] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.362993 3539 ts_tablet_manager.cc:1905] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.363900 3151 consensus_queue.cc:1035] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.375779 3539 ts_tablet_manager.cc:1918] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:43.376792 2405 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_8df1333a0fd4435aaac045cd3dbc1b1a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:43.377063 3601 leader_election.cc:290] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.379772 3539 log.cc:1199] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9f5d19f0216a4f069237e48c62ee7875
I20250905 19:52:43.380342 3539 ts_tablet_manager.cc:1939] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.381834 3539 tablet_replica.cc:331] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.382153 3539 raft_consensus.cc:2241] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.384603 883 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf448f9abc348df8d2d2501420d8f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.384828 877 tablet_service.cc:1515] Processing DeleteTablet for tablet fb72fe6049894527917e1481bcb7b03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:38 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.384858 894 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 with delete_type TABLET_DATA_DELETED (replaced by tablet 350197e134c549198771f4f8337d5dfc) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.384987 881 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 with delete_type TABLET_DATA_DELETED (replaced by tablet 26905425077e4746bccb51431da706c9) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.385136 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 with delete_type TABLET_DATA_DELETED (replaced by tablet f1071206a7cd418f81bf2ce4eaab5237) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.385190 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 0039b8f0ba704907b2820c9e3a69cd73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.385363 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d with delete_type TABLET_DATA_DELETED (replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.385535 3539 raft_consensus.cc:2270] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.385589 882 tablet_service.cc:1515] Processing DeleteTablet for tablet be1315824fd749049a62af3e6a113b9a with delete_type TABLET_DATA_DELETED (replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.386256 3539 ts_tablet_manager.cc:1905] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.387291 3557 tablet_bootstrap.cc:492] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:43.381150 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]) successfully deleted
W20250905 19:52:43.389108 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 89a73e984cbb4268b563f1ca01f4c65d because the tablet was not found. No further retry: Not found: Tablet not found: 89a73e984cbb4268b563f1ca01f4c65d
I20250905 19:52:43.392143 3557 ts_tablet_manager.cc:1397] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.371s user 0.006s sys 0.001s
W20250905 19:52:43.394593 2405 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8df1333a0fd4435aaac045cd3dbc1b1a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:43.396107 3546 raft_consensus.cc:491] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.396247 3546 raft_consensus.cc:513] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.396908 3557 raft_consensus.cc:357] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.397127 3557 raft_consensus.cc:383] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.397156 3546 leader_election.cc:290] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.397186 3557 raft_consensus.cc:738] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.397403 3557 consensus_queue.cc:260] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.398517 1170 tablet_bootstrap.cc:492] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.398520 712 leader_election.cc:304] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:43.398715 1170 ts_tablet_manager.cc:1397] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.208s user 0.017s sys 0.036s
I20250905 19:52:43.398933 3546 raft_consensus.cc:2802] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:43.399003 3546 raft_consensus.cc:491] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:43.399061 3546 raft_consensus.cc:3058] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.399549 1170 raft_consensus.cc:357] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.399740 1170 raft_consensus.cc:383] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.399801 1170 raft_consensus.cc:738] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.400050 1170 consensus_queue.cc:260] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.400477 1170 ts_tablet_manager.cc:1428] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20250905 19:52:43.400729 1170 tablet_bootstrap.cc:492] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:43.401806 1169 tablet_bootstrap.cc:492] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:43.402021 1169 ts_tablet_manager.cc:1397] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.263s user 0.006s sys 0.002s
I20250905 19:52:43.402595 3546 raft_consensus.cc:513] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.402763 1169 raft_consensus.cc:357] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.402977 1169 raft_consensus.cc:383] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.403065 1169 raft_consensus.cc:738] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.403321 1169 consensus_queue.cc:260] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.403854 1169 ts_tablet_manager.cc:1428] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:43.403856 3546 leader_election.cc:290] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.407521 3539 ts_tablet_manager.cc:1918] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:43.408035 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:43.408526 3539 log.cc:1199] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/421b28d047124222ae9cbc603ec22cee
I20250905 19:52:43.408684 774 raft_consensus.cc:2391] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in current term 1: Already voted for candidate f21c498a0d964e8fbebfd1b3cb1916fe in this term.
I20250905 19:52:43.408893 3539 ts_tablet_manager.cc:1939] T 421b28d047124222ae9cbc603ec22cee P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.409233 1170 tablet_bootstrap.cc:654] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.410987 3539 tablet_replica.cc:331] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.411211 3539 raft_consensus.cc:2241] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.411322 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 421b28d047124222ae9cbc603ec22cee (table default.Table_8b6a52346ae4431c93ff61ccda0c25fc [id=f891b3836bac4a919165431ddff7525a]) successfully deleted
I20250905 19:52:43.411379 3539 raft_consensus.cc:2270] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.411995 3539 ts_tablet_manager.cc:1905] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.413609 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:43.423569 986 leader_election.cc:304] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:43.424373 3601 raft_consensus.cc:2802] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:43.424546 3601 raft_consensus.cc:491] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:43.424716 3601 raft_consensus.cc:3058] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.429299 2342 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda
I20250905 19:52:43.429953 2342 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34736 (request call id 79) took 1186 ms. Trace:
I20250905 19:52:43.430076 2342 rpcz_store.cc:276] 0905 19:52:42.243575 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.044480 (+800905us) service_pool.cc:225] Handling call
0905 19:52:43.045016 (+ 536us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:43.045169 (+ 153us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:43.045233 (+ 64us) catalog_manager.cc:2799] Locking tablets
0905 19:52:43.258381 (+213148us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:43.429117 (+170736us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:43.429206 (+ 89us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:43.429243 (+ 37us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:43.429891 (+ 648us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.260695 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.260839 (+ 144us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.260845 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:43.260848 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:43.260959 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.260971 (+ 12us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.260976 (+ 5us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:43.261091 (+ 115us) tablet.cc:699] Row locks acquired
0905 19:52:43.261096 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:43.261132 (+ 36us) write_op.cc:270] Start()
0905 19:52:43.261173 (+ 41us) write_op.cc:276] Timestamp: P: 1757101963261114 usec, L: 0
0905 19:52:43.261176 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.262110 (+ 934us) log.cc:844] Serialized 759 byte log entry
0905 19:52:43.427926 (+165816us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.428698 (+ 772us) write_op.cc:301] APPLY: starting
0905 19:52:43.428708 (+ 10us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.428740 (+ 32us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.428764 (+ 24us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.428826 (+ 62us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.428852 (+ 26us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:43.428859 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:43.428920 (+ 61us) log.cc:844] Serialized 29 byte log entry
0905 19:52:43.428963 (+ 43us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.428976 (+ 13us) write_op.cc:454] Released schema lock
0905 19:52:43.428981 (+ 5us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"child_traces":[["op",{"apply.queue_time_us":738,"apply.run_cpu_time_us":351,"apply.run_wall_time_us":351,"num_ops":2,"prepare.queue_time_us":1590,"prepare.run_cpu_time_us":798,"prepare.run_wall_time_us":1865,"raft.queue_time_us":36,"raft.run_cpu_time_us":4,"raft.run_wall_time_us":4,"replication_time_us":166734,"threads_started":1}]]}
I20250905 19:52:43.432554 2392 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_a0475758d8b449db8b0d9338b8d2f0f1 [id=22c178eea85f4048b10fb58b135663d6] alter complete (version 1)
I20250905 19:52:43.433521 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.452874 3547 ts_tablet_manager.cc:1918] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:43.453102 3547 log.cc:1199] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/b3ea2e2059a4425892fa3c0757e69e4c
I20250905 19:52:43.453620 3547 ts_tablet_manager.cc:1939] T b3ea2e2059a4425892fa3c0757e69e4c P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:43.459800 3539 ts_tablet_manager.cc:1918] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:43.460068 3539 log.cc:1199] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/fd1280fefc3e43db9928567a734644f1
I20250905 19:52:43.460465 3539 ts_tablet_manager.cc:1939] T fd1280fefc3e43db9928567a734644f1 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.460920 3539 tablet_replica.cc:331] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.461382 3539 raft_consensus.cc:2241] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.461585 3539 raft_consensus.cc:2270] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.461738 1170 tablet_bootstrap.cc:492] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.461862 1170 ts_tablet_manager.cc:1397] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.061s user 0.009s sys 0.007s
I20250905 19:52:43.462267 3539 ts_tablet_manager.cc:1905] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.462617 1170 raft_consensus.cc:357] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.462803 1170 raft_consensus.cc:383] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.462859 1170 raft_consensus.cc:738] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.463052 1170 consensus_queue.cc:260] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.463089 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet fd1280fefc3e43db9928567a734644f1 (table default.Table_c8afd57df21440b5923aef4dd927ae45 [id=4447588b8d2443e5a561758689326e26]) successfully deleted
I20250905 19:52:43.463528 1170 ts_tablet_manager.cc:1428] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:43.463797 1170 tablet_bootstrap.cc:492] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:43.465515 1170 tablet_bootstrap.cc:654] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.468099 3557 ts_tablet_manager.cc:1428] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.072s user 0.002s sys 0.000s
I20250905 19:52:43.470055 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d1a11777cf2d4ea9a5cdf792f73aff76 (table default.Table_858dac3c36484123a1a9de7c20d0f141 [id=b0ea76b1ad2c4a9e9499aa1bd7b5f21d]) successfully deleted
I20250905 19:52:43.472263 3539 ts_tablet_manager.cc:1918] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:43.474273 3539 log.cc:1199] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/14d5eb8bec074d2c8766fe7b159031e5
I20250905 19:52:43.478844 3539 ts_tablet_manager.cc:1939] T 14d5eb8bec074d2c8766fe7b159031e5 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.481144 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 14d5eb8bec074d2c8766fe7b159031e5 (table default.Table_b21bb0be9ae1435c8929de3f6b662d8c [id=a9a1d004ba9f4415b2fb958c789d66ba]) successfully deleted
I20250905 19:52:43.496069 3557 tablet_bootstrap.cc:492] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:43.499428 848 consensus_peers.cc:489] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e58bbb6575fd4437b242032a281914ea. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:43.504700 3610 raft_consensus.cc:491] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.505183 3610 raft_consensus.cc:513] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.505893 3610 leader_election.cc:290] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.507362 3539 tablet_replica.cc:331] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.508950 3539 raft_consensus.cc:2241] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:43.495414 3709 tablet.cc:1722] T c870e689af254fb0acb15330577e5c42 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.509407 3557 tablet_bootstrap.cc:654] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.519522 3557 tablet_bootstrap.cc:492] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:43.522850 3557 ts_tablet_manager.cc:1397] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20250905 19:52:43.527518 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a104b1c8ce14fa2bf101e1b34183cfa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.541898 1053 raft_consensus.cc:2466] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:43.528395 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ca3fc7055c44df282285806ea151e51" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.539899 1041 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.544365 2372 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet af5748bea3bd4f51a0d6c28a1d9044a0
I20250905 19:52:43.544857 2372 catalog_manager.cc:6652] ReplaceTablet: tablet af5748bea3bd4f51a0d6c28a1d9044a0 deleted and replaced by tablet 72cad9c309594b8dbe2dd3bab9e1482e
I20250905 19:52:43.545099 2372 master_service.cc:948] ReplaceTablet: received request to replace tablet 6e15350150db413fa4f100171280f46e from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:43.547900 1169 tablet_bootstrap.cc:492] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:43.555413 849 leader_election.cc:304] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.555707 3641 raft_consensus.cc:2802] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:43.555783 3641 raft_consensus.cc:491] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:43.555840 3641 raft_consensus.cc:3058] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.558972 3641 raft_consensus.cc:513] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.559831 3641 leader_election.cc:290] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
W20250905 19:52:43.560555 849 leader_election.cc:343] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.560683 849 leader_election.cc:304] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:43.560882 3579 raft_consensus.cc:2747] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:43.562585 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.562775 909 raft_consensus.cc:2466] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:43.563211 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:43.563352 909 raft_consensus.cc:3058] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.566646 909 raft_consensus.cc:2466] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:43.567463 909 raft_consensus.cc:1273] T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:52:43.568339 3677 tablet.cc:1722] T c870e689af254fb0acb15330577e5c42 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.576774 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9f5d19f0216a4f069237e48c62ee7875" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.577138 907 raft_consensus.cc:2391] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:43.579257 3539 pending_rounds.cc:70] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Trying to abort 1 pending ops.
I20250905 19:52:43.579331 3539 pending_rounds.cc:77] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7197089635345981440 op_type: NO_OP noop_request { }
I20250905 19:52:43.579439 3539 raft_consensus.cc:2887] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20250905 19:52:43.579525 3539 raft_consensus.cc:2270] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.580170 744 tablet_service.cc:1515] Processing DeleteTablet for tablet af5748bea3bd4f51a0d6c28a1d9044a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 72cad9c309594b8dbe2dd3bab9e1482e) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.580592 745 tablet_service.cc:1468] Processing CreateTablet for tablet fe4cc17bdb6d4af8b323de5116aec179 (DEFAULT_TABLE table=default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.581012 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe4cc17bdb6d4af8b323de5116aec179. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.581867 753 tablet_service.cc:1468] Processing CreateTablet for tablet 167d73b0454645098025ee776447f9a2 (DEFAULT_TABLE table=default.Table_58d960189f5d44cf9f5b6e8a7909461a [id=d525d91ded8f427481dfbaf427138a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.582221 753 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167d73b0454645098025ee776447f9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.585104 3539 ts_tablet_manager.cc:1905] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:43.578723 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet dcf448f9abc348df8d2d2501420d8f45 because the tablet was not found. No further retry: Not found: Tablet not found: dcf448f9abc348df8d2d2501420d8f45
I20250905 19:52:43.592022 2340 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 607c8a8848304491bbd4a7971f96322f
I20250905 19:52:43.598587 3666 raft_consensus.cc:491] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.598757 3666 raft_consensus.cc:513] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.599169 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:43.599351 3666 leader_election.cc:290] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 835daf30e4324782848290be181ae78f (127.0.104.2:35993), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.599581 773 raft_consensus.cc:3058] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.601634 1169 tablet_bootstrap.cc:654] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.602941 1046 raft_consensus.cc:1273] T 96aaaf6d94594861b545f831f77dcb55 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 2. (index mismatch)
I20250905 19:52:43.611706 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c8a8848304491bbd4a7971f96322f with delete_type TABLET_DATA_DELETED (replaced by tablet 882ba37668504da4a26b0ebf92e6a21b) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.612390 3557 raft_consensus.cc:357] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.612764 3557 raft_consensus.cc:383] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.613073 3557 raft_consensus.cc:738] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.613528 3557 consensus_queue.cc:260] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.614724 3557 ts_tablet_manager.cc:1428] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20250905 19:52:43.615300 3557 tablet_bootstrap.cc:492] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:43.616585 2340 catalog_manager.cc:6652] ReplaceTablet: tablet 607c8a8848304491bbd4a7971f96322f deleted and replaced by tablet 882ba37668504da4a26b0ebf92e6a21b
I20250905 19:52:43.617432 2340 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_2df584b8fa57457bb948319fd6619a7a" } modify_external_catalogs: true
I20250905 19:52:43.617729 2340 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_2df584b8fa57457bb948319fd6619a7a" } modify_external_catalogs: true
I20250905 19:52:43.629568 3557 tablet_bootstrap.cc:654] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.442533 3541 ts_tablet_manager.cc:1918] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.466145 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:43.630306 905 raft_consensus.cc:3058] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.633375 905 raft_consensus.cc:2466] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:43.634249 3541 log.cc:1199] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/5a59d7fb5dc54b479d965282b1c7d24b
I20250905 19:52:43.634725 3541 ts_tablet_manager.cc:1939] T 5a59d7fb5dc54b479d965282b1c7d24b P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:43.636785 3547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50082 (request call id 41) took 1124 ms. Trace:
I20250905 19:52:43.637180 3547 rpcz_store.cc:276] 0905 19:52:42.512560 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.512609 (+ 49us) service_pool.cc:225] Handling call
0905 19:52:43.144477 (+631868us) ts_tablet_manager.cc:1145] Deleting tablet b3ea2e2059a4425892fa3c0757e69e4c
0905 19:52:43.144490 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:43.320495 (+176005us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:43.452732 (+132237us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:43.636763 (+184031us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":631661}
I20250905 19:52:43.638002 3547 tablet_replica.cc:331] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:43.639122 3547 raft_consensus.cc:2241] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:43.639768 909 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1110897f9ec14741b9b882b6e09081ab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.664247 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 18) took 1932 ms. Trace:
I20250905 19:52:43.664826 3541 rpcz_store.cc:276] 0905 19:52:41.732252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:41.732300 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:43.272117 (+1539817us) ts_tablet_manager.cc:1145] Deleting tablet 5a59d7fb5dc54b479d965282b1c7d24b
0905 19:52:43.272132 (+ 15us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:43.343163 (+ 71031us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:43.442380 (+ 99217us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:43.664215 (+221835us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1537452}
I20250905 19:52:43.673511 2372 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 6e15350150db413fa4f100171280f46e
I20250905 19:52:43.674113 2372 catalog_manager.cc:6652] ReplaceTablet: tablet 6e15350150db413fa4f100171280f46e deleted and replaced by tablet 6a7e40e3ce194bbab8e460ac11282f0f
I20250905 19:52:43.676183 2405 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_b4a0d3409e7648df827261b499b9f127" } new_table_name: "default.Table_b43ce2c9b41a4d04878bade05baf1af0" modify_external_catalogs: true
I20250905 19:52:43.678015 2399 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_e113565700b647959f0e06bc381630e6" } modify_external_catalogs: true
I20250905 19:52:43.678194 2399 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_e113565700b647959f0e06bc381630e6" } modify_external_catalogs: true
I20250905 19:52:43.679863 2372 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_e892ac4352f44019ab940683425afff7" } new_table_name: "default.Table_177f75e6deaa454089d998ea0345348b" modify_external_catalogs: true
I20250905 19:52:43.641518 1048 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3b3bcd5a4f9c46e99664f6cbe9167301" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:43.641582 1052 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:43.693532 1048 raft_consensus.cc:2391] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:43.642590 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.694017 3547 raft_consensus.cc:2270] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.695353 3547 ts_tablet_manager.cc:1905] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.541277 3601 raft_consensus.cc:513] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.675287 986 leader_election.cc:304] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:43.691728 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a104b1c8ce14fa2bf101e1b34183cfa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:43.703758 1053 raft_consensus.cc:3058] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:43.704223 3610 raft_consensus.cc:2802] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:43.704376 3610 raft_consensus.cc:695] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Becoming Leader. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Running, Role: LEADER
I20250905 19:52:43.704664 3610 consensus_queue.cc:237] T 3b3bcd5a4f9c46e99664f6cbe9167301 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.707141 1053 raft_consensus.cc:2466] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:43.715229 1041 raft_consensus.cc:2391] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:43.720688 3601 leader_election.cc:290] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.727133 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b with delete_type TABLET_DATA_DELETED (replaced by tablet b23111dc8af94b56af9538f973248f2a) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.727722 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 89a73e984cbb4268b563f1ca01f4c65d with delete_type TABLET_DATA_DELETED (replaced by tablet c4e641ac5b5c49fda12450bbf1fec2b4) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.728157 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet dcf448f9abc348df8d2d2501420d8f45 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.728597 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet fb72fe6049894527917e1481bcb7b03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:38 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.729000 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 087172093fff4c41980dc2daeabb07c3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.729748 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.739097 3536 raft_consensus.cc:491] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.739287 3536 raft_consensus.cc:513] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.740309 3536 leader_election.cc:290] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.742152 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9f953a7490434bc7afa5510c33e79401 (table default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77]). Expected version 1 got 0
I20250905 19:52:43.743570 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:43.744486 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:43.744886 2342 catalog_manager.cc:5582] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:43.746909 2342 catalog_manager.cc:5582] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:43.747982 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:43.748373 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_da24a2b5f46f4c38b7cff7dcb33c3bf3 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 3 got 2
I20250905 19:52:43.749233 2342 catalog_manager.cc:5582] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:43.749672 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 51d92f92940b4952b68becf86661d459 (table default.Table_f055252e328e48ca9402b1265436d635 [id=848f6c88009643beb5a2abcf7d5c35b7]). Expected version 1 got 0
I20250905 19:52:43.749972 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:43.667613 3557 tablet_bootstrap.cc:492] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:43.750515 3557 ts_tablet_manager.cc:1397] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.135s user 0.004s sys 0.000s
I20250905 19:52:43.750595 3541 tablet_replica.cc:331] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:43.750785 3541 raft_consensus.cc:2241] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:43.750702 771 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a104b1c8ce14fa2bf101e1b34183cfa" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:43.750365 2342 catalog_manager.cc:5582] T 10fef848b9b746c5a0de6a8c3ca74f7a P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:43.751159 3541 raft_consensus.cc:2270] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.751793 3541 ts_tablet_manager.cc:1905] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.752022 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 6597c91bec4f4143ab3ed40afe53a35e (table default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e]). Expected version 1 got 0
I20250905 19:52:43.752581 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:43.753099 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:43.753407 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:43.753719 2342 catalog_manager.cc:5582] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:43.754455 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:43.754726 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 3 got 0
I20250905 19:52:43.756040 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 350197e134c549198771f4f8337d5dfc (table default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431 [id=248e84cad5c3471c947c275c89d6fe50]). Expected version 1 got 0
I20250905 19:52:43.756560 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:43.757026 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:43.757323 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet c870e689af254fb0acb15330577e5c42 (table default.Table_5bd287e779f048a584c35879c6f87713 [id=525ecc2d31fc4e999f35b7514ea01c52]). Expected version 1 got 0
I20250905 19:52:43.758306 2342 catalog_manager.cc:5582] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:43.758524 3579 raft_consensus.cc:491] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.758682 3579 raft_consensus.cc:513] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.758780 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:43.759244 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:43.759774 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e15350150db413fa4f100171280f46e with delete_type TABLET_DATA_DELETED (replaced by tablet 6a7e40e3ce194bbab8e460ac11282f0f) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:43.759640 2342 catalog_manager.cc:5582] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } }
I20250905 19:52:43.760315 744 tablet_service.cc:1468] Processing CreateTablet for tablet db2d0ff4c9ae443b908c33c57dbc449c (DEFAULT_TABLE table=default.Table_8acce2d2e1bf4adeaa0f35f0feabb057 [id=9f5bb7098f7d407288b3737c99b51f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.760834 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db2d0ff4c9ae443b908c33c57dbc449c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.760767 2342 catalog_manager.cc:5582] T c3b2fcc62ff0415f8073edb79b844600 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:43.761183 743 tablet_service.cc:1468] Processing CreateTablet for tablet a8bc702efa3e4d27a43536afe8f32d0c (DEFAULT_TABLE table=default.Table_0852970417d84646b80e1c1fd0dea533 [id=f7a5fe9b4af748359a5fd523b4906fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.761575 2342 catalog_manager.cc:5674] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:43.761643 749 tablet_service.cc:1468] Processing CreateTablet for tablet 31bd5b267cc646d68cb6d450c4bea168 (DEFAULT_TABLE table=default.Table_e52d84f8329e4b4db5f19402e0963db3 [id=b9c9c75d671d41c8bc2fbbbfdda35dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.762043 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31bd5b267cc646d68cb6d450c4bea168. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.762161 2342 catalog_manager.cc:5582] T 2642bb409bb4473db034035b1b834739 P 4eb3af79b01d4835bf6bf7b26bf016f5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:43.763887 822 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:43.765045 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.765450 882 tablet_service.cc:1515] Processing DeleteTablet for tablet af5748bea3bd4f51a0d6c28a1d9044a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 72cad9c309594b8dbe2dd3bab9e1482e) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.765985 882 tablet_service.cc:1468] Processing CreateTablet for tablet fe4cc17bdb6d4af8b323de5116aec179 (DEFAULT_TABLE table=default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.766338 882 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe4cc17bdb6d4af8b323de5116aec179. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:43.766853 986 leader_election.cc:343] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 835daf30e4324782848290be181ae78f (127.0.104.2:35993): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.767038 875 tablet_service.cc:1468] Processing CreateTablet for tablet 167d73b0454645098025ee776447f9a2 (DEFAULT_TABLE table=default.Table_58d960189f5d44cf9f5b6e8a7909461a [id=d525d91ded8f427481dfbaf427138a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.767367 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167d73b0454645098025ee776447f9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.767694 3151 consensus_queue.cc:1035] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.768167 881 tablet_service.cc:1468] Processing CreateTablet for tablet db2d0ff4c9ae443b908c33c57dbc449c (DEFAULT_TABLE table=default.Table_8acce2d2e1bf4adeaa0f35f0feabb057 [id=9f5bb7098f7d407288b3737c99b51f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.768487 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db2d0ff4c9ae443b908c33c57dbc449c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.774753 3151 consensus_queue.cc:1035] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.776871 1169 tablet_bootstrap.cc:492] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:43.777096 1169 ts_tablet_manager.cc:1397] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.239s user 0.004s sys 0.000s
I20250905 19:52:43.777396 741 tablet_service.cc:1468] Processing CreateTablet for tablet 7054e9b5b63945e99c39f895b33866c1 (DEFAULT_TABLE table=default.Table_9b1ba150be754450be1e96d29083d126 [id=ea2c0527a2ee4e28837321ffdb1e4bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.777630 739 tablet_service.cc:1468] Processing CreateTablet for tablet 6f82f0297fd348979b3cb4cbad950ca6 (DEFAULT_TABLE table=default.Table_a0373824c7c44e6996ef440d60dc4cca [id=0adf18ac69d04e8d8c6d7a314a74b1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.777731 751 tablet_service.cc:1468] Processing CreateTablet for tablet 72cad9c309594b8dbe2dd3bab9e1482e (DEFAULT_TABLE table=default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.777853 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7054e9b5b63945e99c39f895b33866c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.778059 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f82f0297fd348979b3cb4cbad950ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.778007 1169 raft_consensus.cc:357] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.778162 1169 raft_consensus.cc:383] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.778209 1169 raft_consensus.cc:738] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.778294 751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72cad9c309594b8dbe2dd3bab9e1482e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.778394 1169 consensus_queue.cc:260] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.778561 742 tablet_service.cc:1468] Processing CreateTablet for tablet f1071206a7cd418f81bf2ce4eaab5237 (DEFAULT_TABLE table=default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.778867 742 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1071206a7cd418f81bf2ce4eaab5237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.778862 1169 ts_tablet_manager.cc:1428] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:43.779135 1169 tablet_bootstrap.cc:492] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:43.779192 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8bc702efa3e4d27a43536afe8f32d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.780939 1169 tablet_bootstrap.cc:654] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.782979 738 tablet_service.cc:1468] Processing CreateTablet for tablet 472a5b10a68b433d94032ef3a898e69c (DEFAULT_TABLE table=default.Table_d7da0965c1624d51adff3f0ef965be87 [id=d86c65600bd5428f8e3e0aa7a50bc270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.783378 738 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472a5b10a68b433d94032ef3a898e69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.787380 773 raft_consensus.cc:2466] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:43.787432 740 tablet_service.cc:1468] Processing CreateTablet for tablet 302b7d821d0741bea829ff059529f7f6 (DEFAULT_TABLE table=default.Table_17b10ad086cd460faea85db26ac8dcd3 [id=3490d90adcdd477d80bce844771e85e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.787873 740 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302b7d821d0741bea829ff059529f7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.789786 3749 tablet.cc:1722] T 9f953a7490434bc7afa5510c33e79401 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.790338 3747 tablet.cc:1722] T 6597c91bec4f4143ab3ed40afe53a35e P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.792894 907 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:43.793179 907 raft_consensus.cc:2391] T 30e1ca2bd1f94f748f2022b5669bc07e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in current term 1: Already voted for candidate f21c498a0d964e8fbebfd1b3cb1916fe in this term.
I20250905 19:52:43.794137 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 with delete_type TABLET_DATA_DELETED (replaced by tablet bd3a516f3a364c9ba30fb58b1baed332) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.794564 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet be1315824fd749049a62af3e6a113b9a with delete_type TABLET_DATA_DELETED (replaced by tablet 252afae7396d4d90b4647f4c4cc7b7cf) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.724051 712 leader_election.cc:304] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:43.796411 3666 raft_consensus.cc:2762] T 3b3bcd5a4f9c46e99664f6cbe9167301 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
W20250905 19:52:43.796384 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 72383.7us
I20250905 19:52:43.796670 712 leader_election.cc:304] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:43.796908 3666 raft_consensus.cc:2802] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:43.797677 3546 consensus_queue.cc:1035] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:43.799831 877 tablet_service.cc:1468] Processing CreateTablet for tablet 302b7d821d0741bea829ff059529f7f6 (DEFAULT_TABLE table=default.Table_17b10ad086cd460faea85db26ac8dcd3 [id=3490d90adcdd477d80bce844771e85e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.800325 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302b7d821d0741bea829ff059529f7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.801187 893 tablet_service.cc:1468] Processing CreateTablet for tablet 31bd5b267cc646d68cb6d450c4bea168 (DEFAULT_TABLE table=default.Table_e52d84f8329e4b4db5f19402e0963db3 [id=b9c9c75d671d41c8bc2fbbbfdda35dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.801388 878 tablet_service.cc:1468] Processing CreateTablet for tablet 7054e9b5b63945e99c39f895b33866c1 (DEFAULT_TABLE table=default.Table_9b1ba150be754450be1e96d29083d126 [id=ea2c0527a2ee4e28837321ffdb1e4bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.801503 893 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31bd5b267cc646d68cb6d450c4bea168. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.801771 889 tablet_service.cc:1468] Processing CreateTablet for tablet 72cad9c309594b8dbe2dd3bab9e1482e (DEFAULT_TABLE table=default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.801505 892 tablet_service.cc:1468] Processing CreateTablet for tablet 6f82f0297fd348979b3cb4cbad950ca6 (DEFAULT_TABLE table=default.Table_a0373824c7c44e6996ef440d60dc4cca [id=0adf18ac69d04e8d8c6d7a314a74b1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.806895 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet e67ed7e9e5724de389504701584bd2b6 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.807281 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 with delete_type TABLET_DATA_DELETED (replaced by tablet 350197e134c549198771f4f8337d5dfc) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.807624 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f with delete_type TABLET_DATA_DELETED (replaced by tablet fe00969eb9654976a0efeac2fa7c91d3) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.808007 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 9a81376d4c0b47a9817deae4d34ffb47 with delete_type TABLET_DATA_DELETED (replaced by tablet 26905425077e4746bccb51431da706c9) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.808341 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 0039b8f0ba704907b2820c9e3a69cd73 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.808673 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 with delete_type TABLET_DATA_DELETED (replaced by tablet f1071206a7cd418f81bf2ce4eaab5237) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.809186 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.809494 1021 tablet_service.cc:1515] Processing DeleteTablet for tablet af5748bea3bd4f51a0d6c28a1d9044a0 with delete_type TABLET_DATA_DELETED (replaced by tablet 72cad9c309594b8dbe2dd3bab9e1482e) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.809906 1170 tablet_bootstrap.cc:492] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:43.810046 1170 ts_tablet_manager.cc:1397] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.346s user 0.020s sys 0.024s
I20250905 19:52:43.810055 1021 tablet_service.cc:1468] Processing CreateTablet for tablet fe4cc17bdb6d4af8b323de5116aec179 (DEFAULT_TABLE table=default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.810940 1033 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c8a8848304491bbd4a7971f96322f with delete_type TABLET_DATA_DELETED (replaced by tablet 882ba37668504da4a26b0ebf92e6a21b) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.811551 1020 tablet_service.cc:1468] Processing CreateTablet for tablet 167d73b0454645098025ee776447f9a2 (DEFAULT_TABLE table=default.Table_58d960189f5d44cf9f5b6e8a7909461a [id=d525d91ded8f427481dfbaf427138a4d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.812979 1170 raft_consensus.cc:357] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.813184 1170 raft_consensus.cc:383] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.813242 1170 raft_consensus.cc:738] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.813443 1170 consensus_queue.cc:260] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.813948 1170 ts_tablet_manager.cc:1428] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20250905 19:52:43.814281 1170 tablet_bootstrap.cc:492] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:43.824113 1170 tablet_bootstrap.cc:654] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:43.828909 3666 raft_consensus.cc:695] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:43.829670 3666 consensus_queue.cc:237] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } }
I20250905 19:52:43.854882 1033 tablet_service.cc:1468] Processing CreateTablet for tablet db2d0ff4c9ae443b908c33c57dbc449c (DEFAULT_TABLE table=default.Table_8acce2d2e1bf4adeaa0f35f0feabb057 [id=9f5bb7098f7d407288b3737c99b51f27]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.857729 886 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e15350150db413fa4f100171280f46e with delete_type TABLET_DATA_DELETED (replaced by tablet 6a7e40e3ce194bbab8e460ac11282f0f) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.860921 3579 leader_election.cc:290] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:43.865962 883 tablet_service.cc:1515] Processing DeleteTablet for tablet 607c8a8848304491bbd4a7971f96322f with delete_type TABLET_DATA_DELETED (replaced by tablet 882ba37668504da4a26b0ebf92e6a21b) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:43.866364 876 tablet_service.cc:1468] Processing CreateTablet for tablet f1071206a7cd418f81bf2ce4eaab5237 (DEFAULT_TABLE table=default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.858146 894 tablet_service.cc:1468] Processing CreateTablet for tablet a8bc702efa3e4d27a43536afe8f32d0c (DEFAULT_TABLE table=default.Table_0852970417d84646b80e1c1fd0dea533 [id=f7a5fe9b4af748359a5fd523b4906fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.873185 3557 raft_consensus.cc:357] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.873363 3557 raft_consensus.cc:383] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:43.873407 3557 raft_consensus.cc:738] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:43.873598 3557 consensus_queue.cc:260] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.873987 3539 ts_tablet_manager.cc:1918] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:43.874083 3539 log.cc:1199] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:43.874106 3557 ts_tablet_manager.cc:1428] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.124s user 0.002s sys 0.000s
I20250905 19:52:43.874418 3539 ts_tablet_manager.cc:1939] T edb9379335ca4eb5a32a6cbe85601a1f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:43.874912 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet edb9379335ca4eb5a32a6cbe85601a1f (table default.Table_3a4d5c41d6f44c128dbb1ef09d643d84 [id=607d3914a3fd4d6089284f064436a1a4]) successfully deleted
W20250905 19:52:43.875502 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 5fdc9d9815864aa7bf5ca9a5c4485390 because the tablet was not found. No further retry: Not found: Tablet not found: 5fdc9d9815864aa7bf5ca9a5c4485390
I20250905 19:52:43.876708 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7054e9b5b63945e99c39f895b33866c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.877348 891 tablet_service.cc:1468] Processing CreateTablet for tablet 472a5b10a68b433d94032ef3a898e69c (DEFAULT_TABLE table=default.Table_d7da0965c1624d51adff3f0ef965be87 [id=d86c65600bd5428f8e3e0aa7a50bc270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.877686 891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472a5b10a68b433d94032ef3a898e69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.884652 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f82f0297fd348979b3cb4cbad950ca6. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:43.886426 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet be1315824fd749049a62af3e6a113b9a because the tablet was not found. No further retry: Not found: Tablet not found: be1315824fd749049a62af3e6a113b9a
I20250905 19:52:43.887261 3539 tablet_replica.cc:331] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:43.887543 3539 raft_consensus.cc:2241] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:43.887856 3539 raft_consensus.cc:2270] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:43.893222 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:43.893437 913 raft_consensus.cc:2466] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
W20250905 19:52:43.895773 848 leader_election.cc:343] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:43.900715 894 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8bc702efa3e4d27a43536afe8f32d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.908937 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e15350150db413fa4f100171280f46e with delete_type TABLET_DATA_DELETED (replaced by tablet 6a7e40e3ce194bbab8e460ac11282f0f) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:43.913765 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 302b7d821d0741bea829ff059529f7f6 (DEFAULT_TABLE table=default.Table_17b10ad086cd460faea85db26ac8dcd3 [id=3490d90adcdd477d80bce844771e85e4]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.913866 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 31bd5b267cc646d68cb6d450c4bea168 (DEFAULT_TABLE table=default.Table_e52d84f8329e4b4db5f19402e0963db3 [id=b9c9c75d671d41c8bc2fbbbfdda35dfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.919131 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:43.925998 1031 tablet_service.cc:1468] Processing CreateTablet for tablet a8bc702efa3e4d27a43536afe8f32d0c (DEFAULT_TABLE table=default.Table_0852970417d84646b80e1c1fd0dea533 [id=f7a5fe9b4af748359a5fd523b4906fec]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:43.893934 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98db7e2a9d6d4f4d984388a493295d72" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:43.930197 3677 tablet.cc:1722] T f034b8cb5d8943938cb2b58c33b45494 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:43.935735 3666 raft_consensus.cc:491] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.935918 3666 raft_consensus.cc:513] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.936633 3666 leader_election.cc:290] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.937417 889 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72cad9c309594b8dbe2dd3bab9e1482e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.938010 876 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1071206a7cd418f81bf2ce4eaab5237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.939457 3557 tablet_bootstrap.cc:492] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:43.941346 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 5a59d7fb5dc54b479d965282b1c7d24b (table default.Table_bb2028f0f4b14f999945c905baf2bc71 [id=6e4f953eca71416d824f67e8a4b0127d]) successfully deleted
I20250905 19:52:43.942320 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet c870e689af254fb0acb15330577e5c42 (table default.Table_5bd287e779f048a584c35879c6f87713 [id=525ecc2d31fc4e999f35b7514ea01c52]). Expected version 1 got 0
I20250905 19:52:43.943379 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:43.943868 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 51d92f92940b4952b68becf86661d459 (table default.Table_f055252e328e48ca9402b1265436d635 [id=848f6c88009643beb5a2abcf7d5c35b7]). Expected version 1 got 0
I20250905 19:52:43.944697 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 302b7d821d0741bea829ff059529f7f6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.945361 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:43.945654 1021 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fe4cc17bdb6d4af8b323de5116aec179. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.946115 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:43.946799 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet fd769c4075914d338cbee14c97402aef (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:43.947021 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:43.948069 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 31bd5b267cc646d68cb6d450c4bea168. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.951253 3151 raft_consensus.cc:491] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.951375 3151 raft_consensus.cc:513] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.952103 1033 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet db2d0ff4c9ae443b908c33c57dbc449c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.952152 3151 leader_election.cc:290] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.953222 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 350197e134c549198771f4f8337d5dfc (table default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431 [id=248e84cad5c3471c947c275c89d6fe50]). Expected version 1 got 0
I20250905 19:52:43.953845 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 93694a3c232a48e180085cd490ee5f4d (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:43.954291 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:43.954650 1031 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8bc702efa3e4d27a43536afe8f32d0c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.955076 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:43.955286 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 3 got 0
I20250905 19:52:43.955416 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
W20250905 19:52:43.955125 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 158051us
I20250905 19:52:43.955590 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:43.955847 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:43.957135 3539 ts_tablet_manager.cc:1905] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:43.957602 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 15be6e2e31364705beb09b4a636e3782 (table default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]). Expected version 1 got 0
I20250905 19:52:43.964131 1020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167d73b0454645098025ee776447f9a2. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:43.965952 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_da24a2b5f46f4c38b7cff7dcb33c3bf3 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 3 got 2
I20250905 19:52:43.966646 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:43.967173 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:43.968595 2397 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:43.971308 3546 raft_consensus.cc:491] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:43.971462 3546 raft_consensus.cc:513] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:43.972049 3546 leader_election.cc:290] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:43.981477 3557 tablet_bootstrap.cc:654] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:43.983668 2373 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet cea5667d882b4b57a6fa4e9866641bf6 consensus state has no leader
I20250905 19:52:44.003566 2370 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_fe9af09b80d04739a6df4abf75c2d717 [id=1598bab0cf8f41e0a4f7ee643cfeeaab] alter complete (version 1)
I20250905 19:52:44.008785 2340 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4edfa9a656554e02b6649e8430c215fb
W20250905 19:52:44.010275 848 leader_election.cc:343] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.016173 3709 tablet.cc:1722] T f034b8cb5d8943938cb2b58c33b45494 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.019836 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4edfa9a656554e02b6649e8430c215fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.020215 3546 raft_consensus.cc:491] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.020430 3546 raft_consensus.cc:513] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.021262 3546 leader_election.cc:290] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.025478 2399 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 1e9b7458f96148b08962a9b82223187e
I20250905 19:52:44.026296 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_a6c14ced6a2f4333985310637ee47d5f"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.026979 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a6c14ced6a2f4333985310637ee47d5f in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.028862 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e9b7458f96148b08962a9b82223187e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.032367 3773 tablet.cc:1722] T d011d1ec8f634386872e7ab257de3187 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.043709 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 7054e9b5b63945e99c39f895b33866c1 (DEFAULT_TABLE table=default.Table_9b1ba150be754450be1e96d29083d126 [id=ea2c0527a2ee4e28837321ffdb1e4bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.044206 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7054e9b5b63945e99c39f895b33866c1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:44.045733 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 53978.5us
I20250905 19:52:44.051592 3468 raft_consensus.cc:491] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.051779 3468 raft_consensus.cc:513] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.053035 1025 tablet_service.cc:1468] Processing CreateTablet for tablet 72cad9c309594b8dbe2dd3bab9e1482e (DEFAULT_TABLE table=default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.053175 1029 tablet_service.cc:1468] Processing CreateTablet for tablet f1071206a7cd418f81bf2ce4eaab5237 (DEFAULT_TABLE table=default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.053436 1025 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 72cad9c309594b8dbe2dd3bab9e1482e. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.053611 1029 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f1071206a7cd418f81bf2ce4eaab5237. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.060334 1027 tablet_service.cc:1468] Processing CreateTablet for tablet 6f82f0297fd348979b3cb4cbad950ca6 (DEFAULT_TABLE table=default.Table_a0373824c7c44e6996ef440d60dc4cca [id=0adf18ac69d04e8d8c6d7a314a74b1e0]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.060761 1027 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6f82f0297fd348979b3cb4cbad950ca6. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.060706 1024 tablet_service.cc:1468] Processing CreateTablet for tablet 472a5b10a68b433d94032ef3a898e69c (DEFAULT_TABLE table=default.Table_d7da0965c1624d51adff3f0ef965be87 [id=d86c65600bd5428f8e3e0aa7a50bc270]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.060617 2342 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_7ccec4c81ed34b1b9fa8ae78c4a47b9d [id=ed7ed5bb46a84eb2bc77d1f948592244] alter complete (version 1)
I20250905 19:52:44.061051 1024 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 472a5b10a68b433d94032ef3a898e69c. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.068933 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.069200 902 raft_consensus.cc:2391] T 298a325f5e4645beac39eb59ddc0566d P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:44.074774 3539 ts_tablet_manager.cc:1918] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:44.076931 3539 log.cc:1199] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/3742e3b46ed24b9082fa3ba5a8e5ee29
I20250905 19:52:44.077503 3539 ts_tablet_manager.cc:1939] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:44.078537 2405 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 882ba37668504da4a26b0ebf92e6a21b has no consensus state
I20250905 19:52:44.082249 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 3742e3b46ed24b9082fa3ba5a8e5ee29 (table default.Table_e34bbf7d7da54ba3be25ddf22622b103 [id=9162fdb871f5487683e3bfb71dfabc96]) successfully deleted
I20250905 19:52:44.083155 1031 tablet_service.cc:1515] Processing DeleteTablet for tablet 4edfa9a656554e02b6649e8430c215fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:44.084239 3539 tablet_replica.cc:331] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.084504 1027 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e9b7458f96148b08962a9b82223187e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:50082
I20250905 19:52:44.084815 3539 raft_consensus.cc:2241] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.089349 773 raft_consensus.cc:3058] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.096873 3539 raft_consensus.cc:2270] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.097829 3539 ts_tablet_manager.cc:1905] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.102209 2403 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34736 (request call id 96) took 1147 ms. Trace:
I20250905 19:52:44.102421 2403 rpcz_store.cc:276] 0905 19:52:42.955041 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.306245 (+351204us) service_pool.cc:225] Handling call
0905 19:52:43.306626 (+ 381us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:43.306869 (+ 243us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:43.307044 (+ 175us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:43.307118 (+ 74us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:43.643416 (+336298us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:44.102137 (+458721us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.572395 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.572536 (+ 141us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.572543 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:43.572546 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:43.572668 (+ 122us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.572681 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.572686 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:43.572759 (+ 73us) tablet.cc:699] Row locks acquired
0905 19:52:43.572762 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:43.572804 (+ 42us) write_op.cc:270] Start()
0905 19:52:43.572845 (+ 41us) write_op.cc:276] Timestamp: P: 1757101963572784 usec, L: 0
0905 19:52:43.572849 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.573070 (+ 221us) log.cc:844] Serialized 405 byte log entry
0905 19:52:43.641326 (+ 68256us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.642127 (+ 801us) write_op.cc:301] APPLY: starting
0905 19:52:43.642141 (+ 14us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.642178 (+ 37us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.642205 (+ 27us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.642246 (+ 41us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.642275 (+ 29us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:43.642281 (+ 6us) write_op.cc:312] APPLY: finished
0905 19:52:43.642351 (+ 70us) log.cc:844] Serialized 23 byte log entry
0905 19:52:43.642398 (+ 47us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.642412 (+ 14us) write_op.cc:454] Released schema lock
0905 19:52:43.642417 (+ 5us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:43.642484 (+ 67us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.642489 (+ 5us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":761,"apply.run_cpu_time_us":353,"apply.run_wall_time_us":352,"num_ops":1,"prepare.queue_time_us":264794,"prepare.run_cpu_time_us":802,"prepare.run_wall_time_us":1277,"raft.queue_time_us":85,"raft.run_cpu_time_us":2,"raft.run_wall_time_us":3,"replication_time_us":68459}]]}
I20250905 19:52:44.104696 2403 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_254b0f36231f4cdd9d758ce8f78526c9"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:44.105489 2405 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_d1234b81a24b4cdfbcfe61677e6bb957" } new_table_name: "default.Table_dee756fb8b38424486b728cf9598eb5c" modify_external_catalogs: true
I20250905 19:52:44.110394 2372 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_6c3777b869db4fb5ad429e51e021b814"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.111640 2372 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c3777b869db4fb5ad429e51e021b814 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.112741 2373 rpcz_store.cc:275] Call kudu.master.MasterService.AlterTable from 127.0.0.1:34736 (request call id 95) took 1168 ms. Trace:
I20250905 19:52:44.112932 2373 rpcz_store.cc:276] 0905 19:52:42.944478 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.299156 (+354678us) service_pool.cc:225] Handling call
0905 19:52:43.299554 (+ 398us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:43.300466 (+ 912us) catalog_manager.cc:3683] Apply alter schema
0905 19:52:43.301903 (+ 1437us) catalog_manager.cc:3715] Acquired catalog manager lock
0905 19:52:43.302023 (+ 120us) catalog_manager.cc:3873] Updating metadata on disk
0905 19:52:43.643324 (+341301us) catalog_manager.cc:3904] Committing alterations to in-memory state
0905 19:52:44.112699 (+469375us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.452309 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.452449 (+ 140us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.452458 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:43.452462 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:43.452605 (+ 143us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.452622 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.452630 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:43.466301 (+ 13671us) tablet.cc:699] Row locks acquired
0905 19:52:43.466307 (+ 6us) write_op.cc:260] PREPARE: finished
0905 19:52:43.466366 (+ 59us) write_op.cc:270] Start()
0905 19:52:43.466420 (+ 54us) write_op.cc:276] Timestamp: P: 1757101963466341 usec, L: 0
0905 19:52:43.466426 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.553257 (+ 86831us) log.cc:844] Serialized 405 byte log entry
0905 19:52:43.568264 (+ 15007us) spinlock_profiling.cc:243] Waited 86.6 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f73ca03bd62 00007f73cb3e42d5 00007f73cb3e2df9 00007f73cb3eb220 00007f73cb3eb02d 0000000000a774dc 00007f73be5ea14f 00007f73be5ed41b 00007f73be5ed298 0000000000a774dc
0905 19:52:43.641247 (+ 72983us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.641582 (+ 335us) write_op.cc:301] APPLY: starting
0905 19:52:43.641603 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.641651 (+ 48us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.641653 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.641707 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.641743 (+ 36us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:43.641751 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:43.641842 (+ 91us) log.cc:844] Serialized 23 byte log entry
0905 19:52:43.641912 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.641934 (+ 22us) write_op.cc:454] Released schema lock
0905 19:52:43.641942 (+ 8us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:43.642035 (+ 93us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.642039 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":270,"apply.run_cpu_time_us":431,"apply.run_wall_time_us":431,"num_ops":1,"prepare.queue_time_us":149778,"prepare.run_cpu_time_us":2054,"prepare.run_wall_time_us":120031,"raft.queue_time_us":1291,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":6,"replication_time_us":174782,"spinlock_wait_cycles":199191296}]]}
I20250905 19:52:44.114739 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_33b37ace063d463ca1b969346fc26d26"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.115687 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_33b37ace063d463ca1b969346fc26d26 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:44.116907 2403 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_254b0f36231f4cdd9d758ce8f78526c9 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.124755 2405 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_a03f6fcd38264fce9c74b6e716c783af"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.125553 2405 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a03f6fcd38264fce9c74b6e716c783af in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.127070 3539 ts_tablet_manager.cc:1918] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.127192 3539 log.cc:1199] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/9dfd0b2e972c4dd4b5b1b998cd0e0eda
I20250905 19:52:44.127528 3539 ts_tablet_manager.cc:1939] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.129354 3539 tablet_replica.cc:331] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.129736 3539 raft_consensus.cc:2241] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.129959 3539 raft_consensus.cc:2270] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.130523 3539 ts_tablet_manager.cc:1905] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.136874 3539 ts_tablet_manager.cc:1918] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:44.137053 3539 log.cc:1199] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/af5748bea3bd4f51a0d6c28a1d9044a0
I20250905 19:52:44.137451 3539 ts_tablet_manager.cc:1939] T af5748bea3bd4f51a0d6c28a1d9044a0 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.138399 3539 tablet_replica.cc:331] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.138648 3539 raft_consensus.cc:2241] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.138878 3539 raft_consensus.cc:2270] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.139485 3539 ts_tablet_manager.cc:1905] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.146425 3539 ts_tablet_manager.cc:1918] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250905 19:52:44.146595 3539 log.cc:1199] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/607c8a8848304491bbd4a7971f96322f
I20250905 19:52:44.147148 3539 ts_tablet_manager.cc:1939] T 607c8a8848304491bbd4a7971f96322f P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.147508 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 9dfd0b2e972c4dd4b5b1b998cd0e0eda (table default.Table_760e5114bbcd4203a4de6891f9510b66 [id=48c7987d5886464e94039405f0b28a3e]) successfully deleted
I20250905 19:52:44.148260 3539 tablet_replica.cc:331] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.148286 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet af5748bea3bd4f51a0d6c28a1d9044a0 (table default.Table_0c5be34d0e4548f1b16cffbd509c6037 [id=6bd2743ff5874d74bb5a4ab22fbd7d75]) successfully deleted
I20250905 19:52:44.148551 3539 raft_consensus.cc:2241] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.148751 3539 raft_consensus.cc:2270] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.149322 3539 ts_tablet_manager.cc:1905] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.151118 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 607c8a8848304491bbd4a7971f96322f (table default.Table_b43ce2c9b41a4d04878bade05baf1af0 [id=f0398b74058b4ccbac71a2cb2f584a8d]) successfully deleted
I20250905 19:52:44.151862 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1110897f9ec14741b9b882b6e09081ab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:44.152055 773 raft_consensus.cc:2466] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:44.156167 3539 ts_tablet_manager.cc:1918] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.156332 3539 log.cc:1199] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6e15350150db413fa4f100171280f46e
I20250905 19:52:44.156737 3539 ts_tablet_manager.cc:1939] T 6e15350150db413fa4f100171280f46e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.157718 3539 tablet_replica.cc:331] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.157979 3539 raft_consensus.cc:2241] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.158175 3539 raft_consensus.cc:2270] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.158852 3539 ts_tablet_manager.cc:1905] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.160363 2403 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6e5d1701ba1440d7a446f7831e77d23a" } modify_external_catalogs: true
I20250905 19:52:44.160683 2403 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6e5d1701ba1440d7a446f7831e77d23a" } modify_external_catalogs: true
W20250905 19:52:44.162297 2392 catalog_manager.cc:5774] Failed to send AlterTable RPC for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3], current schema version=1): Not found: failed to reset TS proxy: No leader found: Tablet 26905425077e4746bccb51431da706c9 consensus state has no leader
I20250905 19:52:44.162937 910 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.163184 910 raft_consensus.cc:2466] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 0.
I20250905 19:52:44.167023 3468 leader_election.cc:290] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.167480 3539 ts_tablet_manager.cc:1918] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.167690 3539 log.cc:1199] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/0039b8f0ba704907b2820c9e3a69cd73
I20250905 19:52:44.167902 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c200681d4d77425389558e6489fb6032" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:44.168674 3539 ts_tablet_manager.cc:1939] T 0039b8f0ba704907b2820c9e3a69cd73 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:44.169606 848 leader_election.cc:343] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.170989 3539 tablet_replica.cc:331] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.171249 3539 raft_consensus.cc:2241] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.171482 3539 raft_consensus.cc:2270] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.172292 3539 ts_tablet_manager.cc:1905] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.174826 3546 consensus_queue.cc:1035] T 96aaaf6d94594861b545f831f77dcb55 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 2, Last known committed idx: 1, Time since last communication: 0.000s
W20250905 19:52:44.175720 711 leader_election.cc:343] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Not found: Tablet not found: 9f5d19f0216a4f069237e48c62ee7875
I20250905 19:52:44.175868 711 leader_election.cc:304] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
W20250905 19:52:44.176056 711 raft_consensus.cc:2684] T 9f5d19f0216a4f069237e48c62ee7875 P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:44.177193 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]) successfully deleted
I20250905 19:52:44.180372 3539 ts_tablet_manager.cc:1918] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:44.180580 3539 log.cc:1199] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/4edfa9a656554e02b6649e8430c215fb
I20250905 19:52:44.181031 3539 ts_tablet_manager.cc:1939] T 4edfa9a656554e02b6649e8430c215fb P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.182124 3539 tablet_replica.cc:331] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.182369 3539 raft_consensus.cc:2241] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.182579 3539 raft_consensus.cc:2270] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.183161 3539 ts_tablet_manager.cc:1905] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.189855 3539 ts_tablet_manager.cc:1918] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.189975 1170 tablet_bootstrap.cc:492] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.190119 3539 log.cc:1199] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/1e9b7458f96148b08962a9b82223187e
I20250905 19:52:44.190799 905 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe00969eb9654976a0efeac2fa7c91d3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.191043 905 raft_consensus.cc:2466] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:44.190183 1170 ts_tablet_manager.cc:1397] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.376s user 0.008s sys 0.020s
I20250905 19:52:44.190799 3767 tablet.cc:1722] T d011d1ec8f634386872e7ab257de3187 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250905 19:52:44.194135 848 consensus_peers.cc:489] T 0039b8f0ba704907b2820c9e3a69cd73 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 0039b8f0ba704907b2820c9e3a69cd73. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.195889 1170 raft_consensus.cc:357] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.198083 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6e15350150db413fa4f100171280f46e (table default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]) successfully deleted
I20250905 19:52:44.199509 1170 raft_consensus.cc:383] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.200194 3539 ts_tablet_manager.cc:1939] T 1e9b7458f96148b08962a9b82223187e P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.200682 1170 raft_consensus.cc:738] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.204267 1170 consensus_queue.cc:260] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.204910 1170 ts_tablet_manager.cc:1428] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.013s user 0.002s sys 0.000s
I20250905 19:52:44.205411 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 0039b8f0ba704907b2820c9e3a69cd73 (table default.Table_6fc6ab8b63df4d2684a474b096058aee [id=284cd828a56b45eb8526bd5d84f2dc2a]) successfully deleted
I20250905 19:52:44.205745 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 4edfa9a656554e02b6649e8430c215fb (table default.Table_2df584b8fa57457bb948319fd6619a7a [id=859765611d234a4ba15dd451416fd2e9]) successfully deleted
I20250905 19:52:44.210712 1169 tablet_bootstrap.cc:492] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:44.210969 1169 ts_tablet_manager.cc:1397] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.432s user 0.004s sys 0.000s
W20250905 19:52:44.211890 2392 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.2:55401 (request call id 78) took 1978 ms (client timeout 1000 ms). Trace:
I20250905 19:52:44.205971 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 1e9b7458f96148b08962a9b82223187e (table default.Table_e113565700b647959f0e06bc381630e6 [id=8e37d62dd8084f16884bd92266ca1311]) successfully deleted
I20250905 19:52:44.213687 2342 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_e08447fc15f5401eb931185a79ad62c8 [id=b3f6e906472a4471be3c9bdd0a25b96d] alter complete (version 1)
I20250905 19:52:44.217682 3547 ts_tablet_manager.cc:1918] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.217922 3547 log.cc:1199] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/4a9afd0fd4ad4ea0afcab9d7291a76bb
I20250905 19:52:44.229138 1170 tablet_bootstrap.cc:492] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.235299 1170 tablet_bootstrap.cc:654] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.243021 1170 tablet_bootstrap.cc:492] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.243189 1170 ts_tablet_manager.cc:1397] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.014s user 0.002s sys 0.001s
I20250905 19:52:44.243364 903 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e3e1732bff14d49ade14fbfa6a9784c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.243577 903 raft_consensus.cc:2466] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:44.244045 1170 raft_consensus.cc:357] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:44.244391 848 consensus_peers.cc:489] T 9dfd0b2e972c4dd4b5b1b998cd0e0eda P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9dfd0b2e972c4dd4b5b1b998cd0e0eda. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:44.220894 2392 rpcz_store.cc:269] 0905 19:52:42.233869 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.258486 (+ 24617us) service_pool.cc:225] Handling call
0905 19:52:44.211840 (+1953354us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.253348 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.253496 (+ 148us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.253503 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:43.253505 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:43.253613 (+ 108us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.253626 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.253630 (+ 4us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:43.253721 (+ 91us) tablet.cc:699] Row locks acquired
0905 19:52:43.253724 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:43.253759 (+ 35us) write_op.cc:270] Start()
0905 19:52:43.253792 (+ 33us) write_op.cc:276] Timestamp: P: 1757101963253742 usec, L: 0
0905 19:52:43.253795 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.253975 (+ 180us) log.cc:844] Serialized 356 byte log entry
0905 19:52:43.427844 (+173869us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.428141 (+ 297us) write_op.cc:301] APPLY: starting
0905 19:52:43.428161 (+ 20us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.428206 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.428238 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.428309 (+ 71us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.428348 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:43.428356 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:43.428445 (+ 89us) log.cc:844] Serialized 23 byte log entry
0905 19:52:43.428515 (+ 70us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.428535 (+ 20us) write_op.cc:454] Released schema lock
0905 19:52:43.428542 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:43.428621 (+ 79us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.428624 (+ 3us) write_op.cc:454] Released schema lock
Metrics: {"mutex_wait_us":115809,"reported_tablets":162,"child_traces":[["op",{"apply.queue_time_us":224,"apply.run_cpu_time_us":476,"apply.run_wall_time_us":477,"num_ops":1,"prepare.queue_time_us":10338,"prepare.run_cpu_time_us":749,"prepare.run_wall_time_us":1988,"raft.queue_time_us":37,"raft.run_cpu_time_us":5,"raft.run_wall_time_us":5,"replication_time_us":174007}]]}
W20250905 19:52:44.252475 3540 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20250905 19:52:44.254904 3540 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20250905 19:52:44.255110 3540 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20250905 19:52:44.259115 2285 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 882ba37668504da4a26b0ebf92e6a21b (table default.Table_b43ce2c9b41a4d04878bade05baf1af0 [id=f0398b74058b4ccbac71a2cb2f584a8d], current schema version=3) failed: Not found: failed to reset TS proxy: No leader found: Tablet 882ba37668504da4a26b0ebf92e6a21b has no consensus state
I20250905 19:52:44.259536 1170 raft_consensus.cc:383] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.259639 1170 raft_consensus.cc:738] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.260560 1170 consensus_queue.cc:260] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.261138 1170 ts_tablet_manager.cc:1428] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20250905 19:52:44.263407 1170 tablet_bootstrap.cc:492] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.264566 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
I20250905 19:52:44.264998 774 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30e1ca2bd1f94f748f2022b5669bc07e" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:44.268164 1170 tablet_bootstrap.cc:654] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:44.272312 848 consensus_peers.cc:489] T 607c8a8848304491bbd4a7971f96322f P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 607c8a8848304491bbd4a7971f96322f. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.274444 3563 raft_consensus.cc:491] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.274585 3563 raft_consensus.cc:513] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.275131 3563 leader_election.cc:290] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.276233 986 leader_election.cc:304] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:44.276494 3274 consensus_queue.cc:1035] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.277154 3601 raft_consensus.cc:2802] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.277248 3601 raft_consensus.cc:491] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.277319 3601 raft_consensus.cc:3058] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.279179 766 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cea5667d882b4b57a6fa4e9866641bf6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:44.279937 848 leader_election.cc:343] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.281492 903 raft_consensus.cc:1273] T 51d92f92940b4952b68becf86661d459 P 835daf30e4324782848290be181ae78f [term 2 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
W20250905 19:52:44.295209 849 consensus_peers.cc:489] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.295630 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 3c824f5591044c4198a51a25661f71e5 with delete_type TABLET_DATA_DELETED (replaced by tablet 00d1d24868204ce09a7f5e40357fec18) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.297008 1169 raft_consensus.cc:357] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.297247 1169 raft_consensus.cc:383] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.297310 1169 raft_consensus.cc:738] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.297585 1169 consensus_queue.cc:260] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.298717 744 tablet_service.cc:1515] Processing DeleteTablet for tablet c69a41cee5ff4f8f9bccaa513c096118 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.298954 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 6ebab2e3698f4f60a770bde9cc019600 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.299064 752 tablet_service.cc:1515] Processing DeleteTablet for tablet fb72fe6049894527917e1481bcb7b03a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:38 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.299144 741 tablet_service.cc:1515] Processing DeleteTablet for tablet 9f5d19f0216a4f069237e48c62ee7875 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.299396 744 tablet_service.cc:1515] Processing DeleteTablet for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:41 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.299873 1169 ts_tablet_manager.cc:1428] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.089s user 0.002s sys 0.000s
I20250905 19:52:44.300197 1169 tablet_bootstrap.cc:492] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
W20250905 19:52:44.300585 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 3c824f5591044c4198a51a25661f71e5 because the tablet was not found. No further retry: Not found: Tablet not found: 3c824f5591044c4198a51a25661f71e5
W20250905 19:52:44.300958 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet c69a41cee5ff4f8f9bccaa513c096118 because the tablet was not found. No further retry: Not found: Tablet not found: c69a41cee5ff4f8f9bccaa513c096118
I20250905 19:52:44.261574 901 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5cf2bc19b043f68838f8a6af6393cd" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.301563 901 raft_consensus.cc:2466] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
W20250905 19:52:44.301954 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 6ebab2e3698f4f60a770bde9cc019600 because the tablet was not found. No further retry: Not found: Tablet not found: 6ebab2e3698f4f60a770bde9cc019600
W20250905 19:52:44.302156 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fb72fe6049894527917e1481bcb7b03a because the tablet was not found. No further retry: Not found: Tablet not found: fb72fe6049894527917e1481bcb7b03a
W20250905 19:52:44.302399 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 9f5d19f0216a4f069237e48c62ee7875 because the tablet was not found. No further retry: Not found: Tablet not found: 9f5d19f0216a4f069237e48c62ee7875
W20250905 19:52:44.302595 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet dc5428f20ca14fdf88a3a52f7b4d1e19 because the tablet was not found. No further retry: Not found: Tablet not found: dc5428f20ca14fdf88a3a52f7b4d1e19
W20250905 19:52:44.302919 3764 log.cc:927] Time spent T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.131s user 0.000s sys 0.000s
I20250905 19:52:44.304386 741 tablet_service.cc:1515] Processing DeleteTablet for tablet e6ac13b19fb6427a9ef35a5160033a54 with delete_type TABLET_DATA_DELETED (replaced by tablet 98db7e2a9d6d4f4d984388a493295d72) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.304457 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 6e3e1732bff14d49ade14fbfa6a9784c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:37950
W20250905 19:52:44.305037 3742 log.cc:927] Time spent T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.132s user 0.000s sys 0.000s
W20250905 19:52:44.305208 3743 log.cc:927] Time spent T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250905 19:52:44.304409 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 421b28d047124222ae9cbc603ec22cee with delete_type TABLET_DATA_DELETED (replaced by tablet 5a5d38136c034b8497045176f749a98d) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.305956 3539 tablet_replica.cc:331] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.306193 3539 raft_consensus.cc:2241] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.306344 3539 raft_consensus.cc:2270] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.306872 3539 ts_tablet_manager.cc:1905] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:44.309929 3777 log.cc:927] Time spent T 96aaaf6d94594861b545f831f77dcb55 P 835daf30e4324782848290be181ae78f: Append to log took a long time: real 0.064s user 0.000s sys 0.000s
I20250905 19:52:44.314177 3677 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:60504 (request call id 75) took 1124 ms. Trace:
I20250905 19:52:44.317808 3539 ts_tablet_manager.cc:1918] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:44.318058 3539 log.cc:1199] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/6e3e1732bff14d49ade14fbfa6a9784c
I20250905 19:52:44.318493 3539 ts_tablet_manager.cc:1939] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
W20250905 19:52:44.320243 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e6ac13b19fb6427a9ef35a5160033a54 because the tablet was not found. No further retry: Not found: Tablet not found: e6ac13b19fb6427a9ef35a5160033a54
I20250905 19:52:44.320586 3547 ts_tablet_manager.cc:1939] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:44.320684 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 6e3e1732bff14d49ade14fbfa6a9784c (table default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d]) successfully deleted
W20250905 19:52:44.320935 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 421b28d047124222ae9cbc603ec22cee because the tablet was not found. No further retry: Not found: Tablet not found: 421b28d047124222ae9cbc603ec22cee
I20250905 19:52:44.321009 1170 tablet_bootstrap.cc:492] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.321182 1170 ts_tablet_manager.cc:1397] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.058s user 0.002s sys 0.009s
I20250905 19:52:44.321964 1170 raft_consensus.cc:357] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.322173 1170 raft_consensus.cc:383] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.322227 1170 raft_consensus.cc:738] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.322435 1170 consensus_queue.cc:260] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.323916 3557 tablet_bootstrap.cc:492] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.324196 3557 ts_tablet_manager.cc:1397] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.385s user 0.004s sys 0.011s
I20250905 19:52:44.325373 752 tablet_service.cc:1515] Processing DeleteTablet for tablet edb9379335ca4eb5a32a6cbe85601a1f with delete_type TABLET_DATA_DELETED (replaced by tablet fe00969eb9654976a0efeac2fa7c91d3) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.326216 752 tablet_service.cc:1515] Processing DeleteTablet for tablet fd1280fefc3e43db9928567a734644f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:22 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.326992 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb with delete_type TABLET_DATA_DELETED (replaced by tablet fe4cc17bdb6d4af8b323de5116aec179) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.327912 752 tablet_service.cc:1515] Processing DeleteTablet for tablet e58bbb6575fd4437b242032a281914ea with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:42 UTC) from {username='slave'} at 127.0.0.1:37950
W20250905 19:52:44.328548 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet edb9379335ca4eb5a32a6cbe85601a1f because the tablet was not found. No further retry: Not found: Tablet not found: edb9379335ca4eb5a32a6cbe85601a1f
I20250905 19:52:44.314267 3677 rpcz_store.cc:276] 0905 19:52:43.380284 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.380334 (+ 50us) service_pool.cc:225] Handling call
0905 19:52:44.314158 (+933824us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.380921 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:43.381034 (+ 113us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:43.381042 (+ 8us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:43.381049 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:43.381086 (+ 37us) alter_schema_op.cc:128] START. Timestamp: 1757101963381062
0905 19:52:43.381089 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.381257 (+ 168us) log.cc:844] Serialized 173 byte log entry
0905 19:52:43.930043 (+548786us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.930166 (+ 123us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:43.930647 (+ 481us) log.cc:844] Serialized 5 byte log entry
0905 19:52:44.313607 (+382960us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.313888 (+ 281us) log.cc:844] Serialized 14 byte log entry
0905 19:52:44.314115 (+ 227us) alter_schema_op.cc:73] Released schema lock
0905 19:52:44.314121 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":41,"prepare.queue_time_us":238,"prepare.run_cpu_time_us":8330,"prepare.run_wall_time_us":91895,"raft.queue_time_us":392,"raft.run_cpu_time_us":473,"raft.run_wall_time_us":472,"replication_time_us":548910,"thread_start_us":90729,"threads_started":1,"wal-append.queue_time_us":90327}]]}
W20250905 19:52:44.329486 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet fd1280fefc3e43db9928567a734644f1 because the tablet was not found. No further retry: Not found: Tablet not found: fd1280fefc3e43db9928567a734644f1
W20250905 19:52:44.329684 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb because the tablet was not found. No further retry: Not found: Tablet not found: 4a9afd0fd4ad4ea0afcab9d7291a76bb
I20250905 19:52:44.330368 1170 ts_tablet_manager.cc:1428] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.001s sys 0.001s
I20250905 19:52:44.330860 1170 tablet_bootstrap.cc:492] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:44.331370 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet e58bbb6575fd4437b242032a281914ea because the tablet was not found. No further retry: Not found: Tablet not found: e58bbb6575fd4437b242032a281914ea
I20250905 19:52:44.333029 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 14d5eb8bec074d2c8766fe7b159031e5 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:23 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.333091 1170 tablet_bootstrap.cc:654] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:44.336282 2285 catalog_manager.cc:4895] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): delete failed for tablet 14d5eb8bec074d2c8766fe7b159031e5 because the tablet was not found. No further retry: Not found: Tablet not found: 14d5eb8bec074d2c8766fe7b159031e5
I20250905 19:52:44.338871 2373 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_4797854b3d9244bd8aac427a5de6985a" } modify_external_catalogs: true
I20250905 19:52:44.339250 2373 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_4797854b3d9244bd8aac427a5de6985a" } modify_external_catalogs: true
I20250905 19:52:44.340354 2405 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_a7fee850ff284e50b2e702f412f360ab"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.341042 2405 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a7fee850ff284e50b2e702f412f360ab in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.341272 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_6c8d37e965d34470a94fab2dcc06f8a5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.341879 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_6c8d37e965d34470a94fab2dcc06f8a5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.342880 2372 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_b24d9f16949145cda6b4f82d78993fa0" } modify_external_catalogs: true
I20250905 19:52:44.343106 2372 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_b24d9f16949145cda6b4f82d78993fa0" } modify_external_catalogs: true
W20250905 19:52:44.346750 3540 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
I20250905 19:52:44.347663 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:44.347931 3540 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20250905 19:52:44.348129 3540 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20250905 19:52:44.348251 3540 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20250905 19:52:44.348760 3547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50082 (request call id 53) took 1297 ms. Trace:
I20250905 19:52:44.348785 2392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_31d60118961a446dbd4a3e75b5afb0cc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:44.348922 3547 rpcz_store.cc:276] 0905 19:52:43.051748 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.052428 (+ 680us) service_pool.cc:225] Handling call
0905 19:52:43.637953 (+585525us) ts_tablet_manager.cc:1145] Deleting tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb
0905 19:52:43.637963 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:43.787797 (+149834us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.217559 (+429762us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.348732 (+131173us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":585366}
W20250905 19:52:44.349391 2392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_31d60118961a446dbd4a3e75b5afb0cc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.349582 3547 tablet_replica.cc:331] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:44.349845 3547 raft_consensus.cc:2241] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.349961 3541 ts_tablet_manager.cc:1918] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.3
I20250905 19:52:44.350086 3541 log.cc:1199] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/087172093fff4c41980dc2daeabb07c3
I20250905 19:52:44.350103 3547 raft_consensus.cc:2270] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.350411 3541 ts_tablet_manager.cc:1939] T 087172093fff4c41980dc2daeabb07c3 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.350653 3547 ts_tablet_manager.cc:1905] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.350674 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 35) took 2301 ms. Trace:
I20250905 19:52:44.350745 3541 rpcz_store.cc:276] 0905 19:52:42.049359 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.049406 (+ 47us) service_pool.cc:225] Handling call
0905 19:52:43.750544 (+1701138us) ts_tablet_manager.cc:1145] Deleting tablet 087172093fff4c41980dc2daeabb07c3
0905 19:52:43.750557 (+ 13us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.166047 (+415490us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.349849 (+183802us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.350660 (+ 811us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1672305}
I20250905 19:52:44.351040 3541 tablet_replica.cc:331] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.361999 3641 raft_consensus.cc:491] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.362195 3641 raft_consensus.cc:513] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.362782 3641 leader_election.cc:290] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.363746 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2edf591963ca46369c29a3fb2bf24bc3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:44.364396 848 leader_election.cc:343] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:44.367769 2370 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.1:45391 (request call id 133) took 1628 ms (client timeout 1000 ms). Trace:
I20250905 19:52:44.373912 2399 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_28de77aaa9fd4ae2b3336689da11bda1"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.374682 2399 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_28de77aaa9fd4ae2b3336689da11bda1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.378942 2405 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_79878218de474236ae52af85d653e417"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.380472 2405 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_79878218de474236ae52af85d653e417 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.384445 2392 master_service.cc:948] ReplaceTablet: received request to replace tablet d59da39e7bcf497a93fb9174328c1814 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:44.387663 3536 consensus_queue.cc:1035] T bd9ae8fd5fdc4b539308268ba9729cd6 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20250905 19:52:44.367933 2370 rpcz_store.cc:269] 0905 19:52:42.739702 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.743295 (+ 3593us) service_pool.cc:225] Handling call
0905 19:52:43.204774 (+461479us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.204786 (+ 12us) write_op.cc:454] Released schema lock
0905 19:52:44.003170 (+798384us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.003180 (+ 10us) write_op.cc:454] Released schema lock
0905 19:52:44.367699 (+364519us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.177131 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.177826 (+ 695us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.177832 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:43.177834 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:43.177940 (+ 106us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.177953 (+ 13us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.177958 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:43.178020 (+ 62us) tablet.cc:699] Row locks acquired
0905 19:52:43.178023 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:43.178055 (+ 32us) write_op.cc:270] Start()
0905 19:52:43.178087 (+ 32us) write_op.cc:276] Timestamp: P: 1757101963178039 usec, L: 0
0905 19:52:43.178091 (+ 4us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.178255 (+ 164us) log.cc:844] Serialized 527 byte log entry
0905 19:52:43.182359 (+ 4104us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.182493 (+ 134us) write_op.cc:301] APPLY: starting
0905 19:52:43.182509 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.182551 (+ 42us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.182576 (+ 25us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.182630 (+ 54us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.182665 (+ 35us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:43.182673 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:43.182761 (+ 88us) log.cc:844] Serialized 23 byte log entry
0905 19:52:43.182824 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.182842 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:43.182848 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:43.992154 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.992346 (+ 192us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.992356 (+ 10us) write_op.cc:435] Acquired schema lock
0905 19:52:43.992360 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:43.992504 (+ 144us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.992524 (+ 20us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.992532 (+ 8us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:43.992615 (+ 83us) tablet.cc:699] Row locks acquired
0905 19:52:43.992619 (+ 4us) write_op.cc:260] PREPARE: finished
0905 19:52:43.992663 (+ 44us) write_op.cc:270] Start()
0905 19:52:43.992701 (+ 38us) write_op.cc:276] Timestamp: P: 1757101963992642 usec, L: 0
0905 19:52:43.992704 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.992976 (+ 272us) log.cc:844] Serialized 356 byte log entry
0905 19:52:43.998629 (+ 5653us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.998801 (+ 172us) write_op.cc:301] APPLY: starting
0905 19:52:43.998822 (+ 21us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.998878 (+ 56us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.998910 (+ 32us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.998996 (+ 86us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.999043 (+ 47us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:43.999055 (+ 12us) write_op.cc:312] APPLY: finished
0905 19:52:43.999162 (+ 107us) log.cc:844] Serialized 23 byte log entry
0905 19:52:43.999233 (+ 71us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.999260 (+ 27us) write_op.cc:454] Released schema lock
0905 19:52:43.999271 (+ 11us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":158,"child_traces":[["op",{"apply.queue_time_us":89,"apply.run_cpu_time_us":407,"apply.run_wall_time_us":407,"num_ops":1,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":690,"prepare.run_wall_time_us":1734,"raft.queue_time_us":56,"raft.run_cpu_time_us":303,"raft.run_wall_time_us":302,"replication_time_us":4239}],["op",{"apply.queue_time_us":109,"apply.run_cpu_time_us":558,"apply.run_wall_time_us":557,"num_ops":1,"prepare.queue_time_us":270,"prepare.run_cpu_time_us":966,"prepare.run_wall_time_us":967,"raft.queue_time_us":1126,"raft.run_cpu_time_us":337,"raft.run_wall_time_us":338,"replication_time_us":5885}]]}
I20250905 19:52:44.394140 2370 master_service.cc:948] ReplaceTablet: received request to replace tablet 160a7814abf9423dbe965c252c20fa7f from {username='slave'} at 127.0.0.1:34736
W20250905 19:52:44.405155 984 consensus_peers.cc:489] T c69a41cee5ff4f8f9bccaa513c096118 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: c69a41cee5ff4f8f9bccaa513c096118. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.406899 3610 consensus_queue.cc:1035] T 00d1d24868204ce09a7f5e40357fec18 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.407080 1170 tablet_bootstrap.cc:492] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.407223 1170 ts_tablet_manager.cc:1397] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.076s user 0.000s sys 0.012s
I20250905 19:52:44.407148 3557 raft_consensus.cc:357] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.407311 3557 raft_consensus.cc:383] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.407371 3557 raft_consensus.cc:738] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.407593 3557 consensus_queue.cc:260] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.408010 1170 raft_consensus.cc:357] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.408166 3557 ts_tablet_manager.cc:1428] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.084s user 0.002s sys 0.000s
I20250905 19:52:44.408185 1170 raft_consensus.cc:383] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.408238 1170 raft_consensus.cc:738] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.408456 3557 tablet_bootstrap.cc:492] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.408463 1170 consensus_queue.cc:260] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.414227 3546 raft_consensus.cc:491] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.414389 3546 raft_consensus.cc:513] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.415062 3546 leader_election.cc:290] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.415897 3557 tablet_bootstrap.cc:654] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.416090 1170 ts_tablet_manager.cc:1428] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.009s user 0.001s sys 0.002s
I20250905 19:52:44.416481 1170 tablet_bootstrap.cc:492] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.418103 3541 raft_consensus.cc:2241] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.418257 3541 raft_consensus.cc:2270] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.418726 3541 ts_tablet_manager.cc:1905] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.421082 1170 tablet_bootstrap.cc:654] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.421228 2399 master_service.cc:948] ReplaceTablet: received request to replace tablet 4a518797d26140f5b9abe41eab415040 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:44.423146 2405 master_service.cc:948] ReplaceTablet: received request to replace tablet 0fb8fe93061848a0b055979bad10f08b from {username='slave'} at 127.0.0.1:34736
W20250905 19:52:44.423576 849 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.3:43385, user_credentials={real_user=slave}} blocked reactor thread for 127497us
I20250905 19:52:44.425848 3541 ts_tablet_manager.cc:1918] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:44.426065 3541 log.cc:1199] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/9f5d19f0216a4f069237e48c62ee7875
I20250905 19:52:44.426045 2392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet d59da39e7bcf497a93fb9174328c1814
W20250905 19:52:44.424036 985 consensus_peers.cc:161] SignalRequest failed: Illegal state: peer closed
I20250905 19:52:44.426528 849 leader_election.cc:304] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:44.426714 3541 ts_tablet_manager.cc:1939] T 9f5d19f0216a4f069237e48c62ee7875 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.426787 3681 raft_consensus.cc:2802] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:44.426895 3681 raft_consensus.cc:695] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:44.427191 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 33) took 2378 ms. Trace:
I20250905 19:52:44.427106 2392 catalog_manager.cc:6652] ReplaceTablet: tablet d59da39e7bcf497a93fb9174328c1814 deleted and replaced by tablet a8ec2b670d224b5799dd2567ee8f5943
I20250905 19:52:44.427158 3681 consensus_queue.cc:237] T 7a104b1c8ce14fa2bf101e1b34183cfa P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.427318 3541 rpcz_store.cc:276] 0905 19:52:42.049155 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.049564 (+ 409us) service_pool.cc:225] Handling call
0905 19:52:44.350999 (+2301435us) ts_tablet_manager.cc:1145] Deleting tablet 9f5d19f0216a4f069237e48c62ee7875
0905 19:52:44.351007 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.422274 (+ 71267us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.425750 (+ 3476us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.427176 (+ 1426us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":66867,"tablet-delete.queue_time_us":2272477}
I20250905 19:52:44.427783 2392 master_service.cc:948] ReplaceTablet: received request to replace tablet 55cecf59d3f24db88bb10b8e12c12f82 from {username='slave'} at 127.0.0.1:34736
I20250905 19:52:44.427928 3541 tablet_replica.cc:331] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.428246 3541 raft_consensus.cc:2241] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.428465 3541 raft_consensus.cc:2270] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.429176 3541 ts_tablet_manager.cc:1905] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:44.430081 984 consensus_peers.cc:489] T 3742e3b46ed24b9082fa3ba5a8e5ee29 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 3742e3b46ed24b9082fa3ba5a8e5ee29. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.431034 752 tablet_service.cc:1515] Processing DeleteTablet for tablet d59da39e7bcf497a93fb9174328c1814 with delete_type TABLET_DATA_DELETED (replaced by tablet a8ec2b670d224b5799dd2567ee8f5943) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.431339 3539 tablet_replica.cc:331] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.431572 3539 raft_consensus.cc:2241] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.431802 3539 raft_consensus.cc:2270] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.432417 3539 ts_tablet_manager.cc:1905] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:44.434873 849 consensus_peers.cc:489] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f -> Peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Couldn't send request to peer 4eb3af79b01d4835bf6bf7b26bf016f5. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: ad0760dfee934aacbd4ab1fb81e116c7. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.439234 2399 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 4a518797d26140f5b9abe41eab415040
I20250905 19:52:44.439851 2399 catalog_manager.cc:6652] ReplaceTablet: tablet 4a518797d26140f5b9abe41eab415040 deleted and replaced by tablet dff4baf04ad34a038a517e8c5142f19b
I20250905 19:52:44.440726 2399 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_70be9db9097645b2b762cf0d447c6032" } modify_external_catalogs: true
I20250905 19:52:44.440919 2399 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_70be9db9097645b2b762cf0d447c6032" } modify_external_catalogs: true
I20250905 19:52:44.441433 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98db7e2a9d6d4f4d984388a493295d72" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:44.441643 1053 raft_consensus.cc:2466] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 0.
I20250905 19:52:44.442265 2392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 55cecf59d3f24db88bb10b8e12c12f82
I20250905 19:52:44.443495 2392 catalog_manager.cc:6652] ReplaceTablet: tablet 55cecf59d3f24db88bb10b8e12c12f82 deleted and replaced by tablet 97d0e923750c4fc89aaeb9a2f9a4a08f
I20250905 19:52:44.444247 2392 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_7afe807a480844efac1cb503fbba94ae" } modify_external_catalogs: true
I20250905 19:52:44.444509 2392 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_7afe807a480844efac1cb503fbba94ae" } modify_external_catalogs: true
I20250905 19:52:44.444906 712 leader_election.cc:304] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.445205 3546 raft_consensus.cc:2802] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.445268 3546 raft_consensus.cc:491] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.445323 3546 raft_consensus.cc:3058] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.448371 744 tablet_service.cc:1515] Processing DeleteTablet for tablet 55cecf59d3f24db88bb10b8e12c12f82 with delete_type TABLET_DATA_DELETED (replaced by tablet 97d0e923750c4fc89aaeb9a2f9a4a08f) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.448581 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a518797d26140f5b9abe41eab415040 with delete_type TABLET_DATA_DELETED (replaced by tablet dff4baf04ad34a038a517e8c5142f19b) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.448740 3557 tablet_bootstrap.cc:492] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.448889 3557 ts_tablet_manager.cc:1397] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.041s user 0.005s sys 0.000s
I20250905 19:52:44.448998 3546 raft_consensus.cc:513] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.450178 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c200681d4d77425389558e6489fb6032" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:44.451453 3546 leader_election.cc:290] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:44.455883 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 179339us
I20250905 19:52:44.452538 3557 raft_consensus.cc:357] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.456682 3557 raft_consensus.cc:383] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.456749 3557 raft_consensus.cc:738] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.456959 3557 consensus_queue.cc:260] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.457309 712 leader_election.cc:304] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.458272 3557 ts_tablet_manager.cc:1428] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250905 19:52:44.458567 3557 tablet_bootstrap.cc:492] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:44.459084 712 raft_consensus.cc:2684] T 6e3e1732bff14d49ade14fbfa6a9784c P f21c498a0d964e8fbebfd1b3cb1916fe: unable to run election callback: Service unavailable: Thread pool token was shut down
I20250905 19:52:44.459190 986 leader_election.cc:304] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters:
I20250905 19:52:44.459549 3536 raft_consensus.cc:2802] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.459687 3536 raft_consensus.cc:491] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.459883 3536 raft_consensus.cc:3058] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.460434 3557 tablet_bootstrap.cc:654] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.462270 752 tablet_service.cc:1468] Processing CreateTablet for tablet 882ba37668504da4a26b0ebf92e6a21b (DEFAULT_TABLE table=default.Table_b43ce2c9b41a4d04878bade05baf1af0 [id=f0398b74058b4ccbac71a2cb2f584a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.462680 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882ba37668504da4a26b0ebf92e6a21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.464773 3546 consensus_queue.cc:1035] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.466096 712 leader_election.cc:304] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.466362 3546 raft_consensus.cc:2802] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.466426 3546 raft_consensus.cc:491] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.466482 3546 raft_consensus.cc:3058] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.466639 744 tablet_service.cc:1468] Processing CreateTablet for tablet 16ac8099de574f39953e3244c92bbbeb (DEFAULT_TABLE table=default.Table_8df1333a0fd4435aaac045cd3dbc1b1a [id=fee1bd51830440debdf8b39c0f79c68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.467036 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16ac8099de574f39953e3244c92bbbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.469353 3546 raft_consensus.cc:513] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.470199 3546 leader_election.cc:290] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.471807 741 tablet_service.cc:1468] Processing CreateTablet for tablet f439e2a456054e0b8c6fe10a76e7b7f1 (DEFAULT_TABLE table=default.Table_3244f6924a9e487d8f33dd11032916ab [id=4f2ea1c223ef41a7b9687c79379d3636]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.472252 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6a7e40e3ce194bbab8e460ac11282f0f (DEFAULT_TABLE table=default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.472609 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7e40e3ce194bbab8e460ac11282f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.473003 1170 tablet_bootstrap.cc:492] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.473155 1170 ts_tablet_manager.cc:1397] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.057s user 0.001s sys 0.001s
I20250905 19:52:44.473325 3541 ts_tablet_manager.cc:1918] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20250905 19:52:44.473438 3541 log.cc:1199] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6e3e1732bff14d49ade14fbfa6a9784c
I20250905 19:52:44.473763 3541 ts_tablet_manager.cc:1939] T 6e3e1732bff14d49ade14fbfa6a9784c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.474030 1170 raft_consensus.cc:357] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.474192 1170 raft_consensus.cc:383] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.474244 1170 raft_consensus.cc:738] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.474541 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 34) took 2425 ms. Trace:
I20250905 19:52:44.474627 3541 rpcz_store.cc:276] 0905 19:52:42.049252 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.049681 (+ 429us) service_pool.cc:225] Handling call
0905 19:52:44.427830 (+2378149us) ts_tablet_manager.cc:1145] Deleting tablet 6e3e1732bff14d49ade14fbfa6a9784c
0905 19:52:44.427839 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.453816 (+ 25977us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.473222 (+ 19406us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.474522 (+ 1300us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2349193}
I20250905 19:52:44.474925 3541 tablet_replica.cc:331] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.475101 3541 raft_consensus.cc:2241] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.475248 3541 raft_consensus.cc:2270] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.476002 3541 ts_tablet_manager.cc:1905] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.480615 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 087172093fff4c41980dc2daeabb07c3 (table default.Table_fd318ab664334e4bb9d57d696f6449d2 [id=d7c0f44965694f77a8eae2ec4dd18a03]) successfully deleted
I20250905 19:52:44.481081 1170 consensus_queue.cc:260] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.481468 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 9f5d19f0216a4f069237e48c62ee7875 (table default.Table_21f0cf32be59451d987d978191525180 [id=58e21ae57fda437e81365a58cfaa0fb1]) successfully deleted
W20250905 19:52:44.489558 831 tablet.cc:2378] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:44.495386 3539 ts_tablet_manager.cc:1918] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.495446 892 tablet_service.cc:1515] Processing DeleteTablet for tablet 4edfa9a656554e02b6649e8430c215fb with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.495675 889 tablet_service.cc:1515] Processing DeleteTablet for tablet 1e9b7458f96148b08962a9b82223187e with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:43 UTC) from {username='slave'} at 127.0.0.1:60504
W20250905 19:52:44.495666 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 65408us
I20250905 19:52:44.495879 876 tablet_service.cc:1515] Processing DeleteTablet for tablet 5a59d7fb5dc54b479d965282b1c7d24b with delete_type TABLET_DATA_DELETED (replaced by tablet b23111dc8af94b56af9538f973248f2a) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.496834 893 tablet_service.cc:1515] Processing DeleteTablet for tablet d59da39e7bcf497a93fb9174328c1814 with delete_type TABLET_DATA_DELETED (replaced by tablet a8ec2b670d224b5799dd2567ee8f5943) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.497118 881 tablet_service.cc:1468] Processing CreateTablet for tablet 16ac8099de574f39953e3244c92bbbeb (DEFAULT_TABLE table=default.Table_8df1333a0fd4435aaac045cd3dbc1b1a [id=fee1bd51830440debdf8b39c0f79c68a]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.497570 882 tablet_service.cc:1515] Processing DeleteTablet for tablet 55cecf59d3f24db88bb10b8e12c12f82 with delete_type TABLET_DATA_DELETED (replaced by tablet 97d0e923750c4fc89aaeb9a2f9a4a08f) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.497601 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16ac8099de574f39953e3244c92bbbeb. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.497895 875 tablet_service.cc:1468] Processing CreateTablet for tablet 882ba37668504da4a26b0ebf92e6a21b (DEFAULT_TABLE table=default.Table_b43ce2c9b41a4d04878bade05baf1af0 [id=f0398b74058b4ccbac71a2cb2f584a8d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.498236 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 882ba37668504da4a26b0ebf92e6a21b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.500037 2370 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 160a7814abf9423dbe965c252c20fa7f
I20250905 19:52:44.500401 876 tablet_service.cc:1515] Processing DeleteTablet for tablet d1a11777cf2d4ea9a5cdf792f73aff76 with delete_type TABLET_DATA_DELETED (replaced by tablet bd3a516f3a364c9ba30fb58b1baed332) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.501442 2370 catalog_manager.cc:6652] ReplaceTablet: tablet 160a7814abf9423dbe965c252c20fa7f deleted and replaced by tablet 734937ccc1cf4a528247cc79f893e5ae
I20250905 19:52:44.501392 877 tablet_service.cc:1468] Processing CreateTablet for tablet f439e2a456054e0b8c6fe10a76e7b7f1 (DEFAULT_TABLE table=default.Table_3244f6924a9e487d8f33dd11032916ab [id=4f2ea1c223ef41a7b9687c79379d3636]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.501818 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f439e2a456054e0b8c6fe10a76e7b7f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.502135 2370 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_8a59f0ac61434a5880164439529876f4" } new_table_name: "default.Table_2f13d41b96a74208927b000350e86150" modify_external_catalogs: true
I20250905 19:52:44.504324 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 4a518797d26140f5b9abe41eab415040 with delete_type TABLET_DATA_DELETED (replaced by tablet dff4baf04ad34a038a517e8c5142f19b) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.504472 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.504627 913 raft_consensus.cc:2466] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:44.505388 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe00969eb9654976a0efeac2fa7c91d3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:44.505517 913 raft_consensus.cc:3058] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.508200 3151 consensus_queue.cc:1035] T 6edbc312fc4947a79e937dbc64d30b37 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.508625 913 raft_consensus.cc:2466] T fe00969eb9654976a0efeac2fa7c91d3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:44.510018 3749 tablet.cc:1722] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.512262 3540 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42539
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_preallocate_segments=false
--leader_failure_max_missed_heartbeat_periods=1
--fs_data_dirs=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
--hive_metastore_notification_log_poll_period_seconds=1
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38687,127.0.104.61:41317,127.0.104.60:34395
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:34395
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=44671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision e2e116a9cb215058aad161061214d11c617cedac
build type FASTDEBUG
built by None at 05 Sep 2025 19:43:22 UTC on 24a791456cd2
build id 7916
ASAN enabled
I20250905 19:52:44.513197 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5cf2bc19b043f68838f8a6af6393cd" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:44.513373 902 raft_consensus.cc:3058] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.516973 902 raft_consensus.cc:2466] T 3c5cf2bc19b043f68838f8a6af6393cd P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:44.518024 3747 tablet.cc:1722] T af10abf0a5b34b3bbfea20e5c2005006 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.527808 3540 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20250905 19:52:44.529630 3747 tablet.cc:1722] T 85c648f799304fa9a94bec665a2dcf97 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:44.532827 3557 tablet_bootstrap.cc:492] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.533008 3557 ts_tablet_manager.cc:1397] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.075s user 0.005s sys 0.000s
I20250905 19:52:44.533974 3557 raft_consensus.cc:357] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.534189 3557 raft_consensus.cc:383] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.534264 3557 raft_consensus.cc:738] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.534489 3557 consensus_queue.cc:260] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.536937 3557 ts_tablet_manager.cc:1428] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:44.537285 3557 tablet_bootstrap.cc:492] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.538815 1169 tablet_bootstrap.cc:654] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.544571 3601 raft_consensus.cc:513] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.545553 3601 leader_election.cc:290] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.546044 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 350197e134c549198771f4f8337d5dfc (table default.Table_6b7f2712ef1d46dcb4f1d4c1a110b431 [id=248e84cad5c3471c947c275c89d6fe50]). Expected version 1 got 0
I20250905 19:52:44.546372 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d495b6ba92e84b588b4780c2896195c0 (table default.Table_fa90bb9b631642f4b6e81bf6253dc642 [id=c31b7e937b6d4adfb299ae8e46330d55]). Expected version 1 got 0
I20250905 19:52:44.546820 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 85c648f799304fa9a94bec665a2dcf97 (table default.Table_da24a2b5f46f4c38b7cff7dcb33c3bf3 [id=7b0316448a484002a5cc2e3db8d4cdf4]). Expected version 3 got 2
I20250905 19:52:44.547087 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:44.547335 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:44.547993 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:44.548267 2340 catalog_manager.cc:5582] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 1 to 2, leader changed from 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3) to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 2 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:44.548786 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 51d92f92940b4952b68becf86661d459 (table default.Table_f055252e328e48ca9402b1265436d635 [id=848f6c88009643beb5a2abcf7d5c35b7]). Expected version 1 got 0
I20250905 19:52:44.549368 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:44.549588 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f6f29ac5e68d4d8da8f3496fefe695ec (table default.Table_234e811e09aa42e2a83b43d09c7d8ee0 [id=1196f4fec5bc41109112409f35a92cb3]). Expected version 1 got 0
I20250905 19:52:44.549908 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet fbba1ad28a614b06afe865c478901a32 (table default.Table_631f9bc61e1442c09b7f9ab3a3e881da [id=0b6f3d59aebb4de0b34652b9f2c8ded7]). Expected version 1 got 0
I20250905 19:52:44.550139 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 03cf686811e041e7a001669b3d86e28a (table default.Table_70be9db9097645b2b762cf0d447c6032 [id=f7d8cd0a6972466a997a8fdd26bccf46]). Expected version 4 got 0
I20250905 19:52:44.553328 960 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:44.554670 3540 file_cache.cc:492] Constructed file cache file cache with capacity 419430
I20250905 19:52:44.555898 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ae6fe4620dda4b1296416e01f8ffee29 (table default.Table_2e5e3cf0f8994adfb2837e2fa148fd98 [id=a42dd4a1d931496f8786c21f55f1c87e]). Expected version 1 got 0
I20250905 19:52:44.558907 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet e10b4e82c4bc429b9a42db39c03a5267 (table default.Table_a11f2d8ae9ab4b419c4e66bdaf4d02f9 [id=6a2d8d9aa80544c6b70397d8f061865b]). Expected version 1 got 0
I20250905 19:52:44.559592 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet d011d1ec8f634386872e7ab257de3187 (table default.Table_f88ff074d1014b9a8a85de3e399792f6 [id=02073fa2a9d44595802b8a3649251d8a]). Expected version 1 got 0
I20250905 19:52:44.563531 3557 tablet_bootstrap.cc:654] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.563818 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet fd769c4075914d338cbee14c97402aef (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]). Expected version 1 got 0
I20250905 19:52:44.564441 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 93694a3c232a48e180085cd490ee5f4d (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]). Expected version 1 got 0
I20250905 19:52:44.565316 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 9ca3fc7055c44df282285806ea151e51 (table default.Table_28f341e829524e1bb47b0bc5a5e549cd [id=89c2ed6b333742bea498aa0da051d60b]). Expected version 5 got 0
I20250905 19:52:44.568112 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6e3e1732bff14d49ade14fbfa6a9784c (table default.Table_ec95e016850b4e4eb1105ac761a31b74 [id=09b582a2cc5f4bdea93ca3c26a4b389d]) successfully deleted
I20250905 19:52:44.568584 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:44.569419 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet c4e641ac5b5c49fda12450bbf1fec2b4 (table default.Table_4832a3f6d70d4a2a9c8e7b2b409f0952 [id=a5ecc77067244a09b1b8b3cea3ed44b7]). Expected version 3 got 0
I20250905 19:52:44.569742 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet cea5667d882b4b57a6fa4e9866641bf6 (table default.Table_6bb624ff162648cda8ec7badc63433ed [id=68581ab2921a42d1931a9edb44fa1ecd]). Expected version 1 got 0
I20250905 19:52:44.570612 2340 catalog_manager.cc:5582] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe reported cstate change: term changed from 0 to 1, leader changed from <none> to f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1). New cstate: current_term: 1 leader_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } health_report { overall_health: UNKNOWN } } }
I20250905 19:52:44.571241 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 15be6e2e31364705beb09b4a636e3782 (table default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]). Expected version 1 got 0
I20250905 19:52:44.571995 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet ea6df8634a314ceabcf3fd004bf7e241 (table default.Table_177f75e6deaa454089d998ea0345348b [id=ea980bda9f2f41c8bf82af3ce10d7151]). Expected version 4 got 3
I20250905 19:52:44.574267 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet f034b8cb5d8943938cb2b58c33b45494 (table default.Table_4c30c89397974e059c9e48f15077d515 [id=9d1528c0d2b1414e9cd5b222124016df]). Expected version 1 got 0
I20250905 19:52:44.574822 2340 catalog_manager.cc:5674] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531) does not have the latest schema for tablet 3315d3e9633e42ea8d7a6b5934cb0438 (table default.Table_dee756fb8b38424486b728cf9598eb5c [id=9c5a9ace09234365a465b92fd6bb7f40]). Expected version 2 got 1
I20250905 19:52:44.575759 849 leader_election.cc:304] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f; no voters: f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:44.576738 3791 raft_consensus.cc:2802] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.576818 3791 raft_consensus.cc:491] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.576881 3791 raft_consensus.cc:3058] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.577286 3536 raft_consensus.cc:513] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.577963 3536 leader_election.cc:290] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.585565 2373 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 74eb2632a6414e52b9ad67f53c182679
I20250905 19:52:44.586627 2373 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_fadffcdfb2fb45f39fe3b3b3df268ce3" } new_table_name: "default.Table_c8d2e8b2928a4c39b58c6961b34b3fea" modify_external_catalogs: true
I20250905 19:52:44.594532 3541 ts_tablet_manager.cc:1918] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.594673 3541 log.cc:1199] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e67ed7e9e5724de389504701584bd2b6
I20250905 19:52:44.595149 3541 ts_tablet_manager.cc:1939] T e67ed7e9e5724de389504701584bd2b6 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.595297 3791 raft_consensus.cc:513] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.595461 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 36) took 2527 ms. Trace:
I20250905 19:52:44.595535 3541 rpcz_store.cc:276] 0905 19:52:42.068236 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.068295 (+ 59us) service_pool.cc:225] Handling call
0905 19:52:44.474884 (+2406589us) ts_tablet_manager.cc:1145] Deleting tablet e67ed7e9e5724de389504701584bd2b6
0905 19:52:44.474892 (+ 8us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.559744 (+ 84852us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.594400 (+ 34656us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.595446 (+ 1046us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2368870}
I20250905 19:52:44.595844 3541 tablet_replica.cc:331] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.596061 3541 raft_consensus.cc:2241] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.596110 3791 leader_election.cc:290] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.596220 3541 raft_consensus.cc:2270] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.596750 3541 ts_tablet_manager.cc:1905] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.598253 2403 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 252afae7396d4d90b4647f4c4cc7b7cf
I20250905 19:52:44.598802 2403 rpcz_store.cc:275] Call kudu.master.MasterService.DeleteTable from 127.0.0.1:34736 (request call id 109) took 1259 ms. Trace:
I20250905 19:52:44.598939 2403 rpcz_store.cc:276] 0905 19:52:43.339470 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:44.160102 (+820632us) service_pool.cc:225] Handling call
0905 19:52:44.160956 (+ 854us) catalog_manager.cc:2368] Looking up, locking, and authorizing table
0905 19:52:44.161238 (+ 282us) catalog_manager.cc:2790] Modifying in-memory table state
0905 19:52:44.161365 (+ 127us) catalog_manager.cc:2799] Locking tablets
0905 19:52:44.222153 (+ 60788us) catalog_manager.cc:2813] Removing table and tablets from system table
0905 19:52:44.358254 (+136101us) catalog_manager.cc:2831] Removing table from by-name map
0905 19:52:44.358315 (+ 61us) catalog_manager.cc:2847] Committing in-memory state
0905 19:52:44.598170 (+239855us) catalog_manager.cc:2851] Aborting table tasks
0905 19:52:44.598759 (+ 589us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:44.223263 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:44.223459 (+ 196us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:44.223468 (+ 9us) write_op.cc:435] Acquired schema lock
0905 19:52:44.223473 (+ 5us) tablet.cc:660] Decoding operations
0905 19:52:44.223646 (+ 173us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:44.223667 (+ 21us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:44.223676 (+ 9us) tablet.cc:683] Acquiring locks for 2 operations
0905 19:52:44.223829 (+ 153us) tablet.cc:699] Row locks acquired
0905 19:52:44.223834 (+ 5us) write_op.cc:260] PREPARE: finished
0905 19:52:44.223901 (+ 67us) write_op.cc:270] Start()
0905 19:52:44.223934 (+ 33us) write_op.cc:276] Timestamp: P: 1757101964223806 usec, L: 0
0905 19:52:44.223937 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:44.344122 (+120185us) log.cc:844] Serialized 725 byte log entry
0905 19:52:44.344328 (+ 206us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 000000000097272a 00007f73ca03bd62 00007f73cb3e42d5 00007f73cb3e2df9 00007f73cb3eb220 00007f73cb3eb02d 0000000000a774dc 00007f73be5ea14f 00007f73be5ed41b 00007f73be5ed298 0000000000a774dc
0905 19:52:44.356247 (+ 11919us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.356464 (+ 217us) write_op.cc:301] APPLY: starting
0905 19:52:44.356480 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:44.356531 (+ 51us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:44.356533 (+ 2us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:44.356613 (+ 80us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:44.356651 (+ 38us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=2
0905 19:52:44.356658 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:44.356736 (+ 78us) log.cc:844] Serialized 29 byte log entry
0905 19:52:44.356802 (+ 66us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.356820 (+ 18us) write_op.cc:454] Released schema lock
0905 19:52:44.356827 (+ 7us) write_op.cc:341] FINISH: Updating metrics
0905 19:52:44.356903 (+ 76us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.356907 (+ 4us) write_op.cc:454] Released schema lock
Metrics: {"child_traces":[["op",{"apply.queue_time_us":169,"apply.run_cpu_time_us":425,"apply.run_wall_time_us":425,"num_ops":2,"prepare.queue_time_us":70,"prepare.run_cpu_time_us":2489,"prepare.run_wall_time_us":121142,"raft.queue_time_us":1994,"raft.run_cpu_time_us":283,"raft.run_wall_time_us":282,"replication_time_us":132277,"spinlock_wait_cycles":275916160}]]}
I20250905 19:52:44.600056 2403 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_fe6d663d69f74f929b1255e176be4ead" } new_table_name: "default.Table_cd27009a0ede4b1aae0edf5a87010337" modify_external_catalogs: true
W20250905 19:52:44.606040 849 leader_election.cc:343] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.606176 849 leader_election.cc:304] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:44.606750 3791 raft_consensus.cc:2747] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:44.608573 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 160a7814abf9423dbe965c252c20fa7f with delete_type TABLET_DATA_DELETED (replaced by tablet 734937ccc1cf4a528247cc79f893e5ae) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.609023 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6a7e40e3ce194bbab8e460ac11282f0f (DEFAULT_TABLE table=default.Table_4786a32c4c014dac926b49908380da96 [id=3a67ff4a6e95484a92512ecb4f1a6cff]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.609131 878 tablet_service.cc:1515] Processing DeleteTablet for tablet 252afae7396d4d90b4647f4c4cc7b7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:44 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.609301 875 tablet_service.cc:1515] Processing DeleteTablet for tablet 74eb2632a6414e52b9ad67f53c182679 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:44 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.609479 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a7e40e3ce194bbab8e460ac11282f0f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.613430 2373 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_4786a32c4c014dac926b49908380da96" } new_table_name: "default.Table_f864acb8d65a42d0aeced6ca942e60a8" modify_external_catalogs: true
I20250905 19:52:44.609025 881 tablet_service.cc:1468] Processing CreateTablet for tablet f6ac6052745e4524a0fad5d1c99dc064 (DEFAULT_TABLE table=default.Table_3dafac26db484c73a9482d5732d1fbf4 [id=51d1ade5c78342bfbb935326a74fe2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.617040 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ac6052745e4524a0fad5d1c99dc064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.619313 3677 tablet.cc:1722] T 3315d3e9633e42ea8d7a6b5934cb0438 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:44.619771 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e67ed7e9e5724de389504701584bd2b6 (table default.Table_bc301cb31d184494ad445294416d0a20 [id=fce3c06d98814e128a2c805c68a1b923]) successfully deleted
W20250905 19:52:44.625490 2373 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet 6a7e40e3ce194bbab8e460ac11282f0f consensus state has no leader
I20250905 19:52:44.626740 2373 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_44734742f0c948af9cce54466ff759c6" } new_table_name: "default.Table_c8cc9c16d77646cf829dda0c8bf49c9b" modify_external_catalogs: true
I20250905 19:52:44.636891 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_8fb82f90dda04873a3b1401d1c0870dc"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.637502 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8fb82f90dda04873a3b1401d1c0870dc in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.639209 2342 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_186206d66e7446e980e368d0716e67df [id=4044052e369843d39980a31b1654628d] alter complete (version 4)
I20250905 19:52:44.642359 3557 tablet_bootstrap.cc:492] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
W20250905 19:52:44.643082 2342 rpcz_store.cc:267] Call kudu.master.MasterService.TSHeartbeat from 127.0.104.3:45433 (request call id 121) took 1236 ms (client timeout 1000 ms). Trace:
I20250905 19:52:44.643003 3557 ts_tablet_manager.cc:1397] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.106s user 0.012s sys 0.001s
W20250905 19:52:44.645970 848 consensus_peers.cc:489] T 1e9b7458f96148b08962a9b82223187e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 1e9b7458f96148b08962a9b82223187e. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:44.643190 2342 rpcz_store.cc:269] 0905 19:52:43.406200 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.430708 (+ 24508us) service_pool.cc:225] Handling call
0905 19:52:43.560682 (+129974us) spinlock_profiling.cc:243] Waited 120 ms on lock 0x616000070954. stack: 00007f73be5a3dc6 00007f73be5a3b80 00007f73bd58b3ad 0000000000972d94 0000000000972d5d 00007f73cd3aa14d 00007f73cd3aa243 00007f73ca069941 00007f73d1bf4a7d 00007f73d1d786f8 00007f73c8bb2c86 00007f73c8bb29b5 00007f73c538d0be 00007f73c538bf9e 00007f73c53921d5 00007f73c5392aff
0905 19:52:43.773387 (+212705us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.773393 (+ 6us) write_op.cc:454] Released schema lock
0905 19:52:44.059621 (+286228us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.059678 (+ 57us) write_op.cc:454] Released schema lock
0905 19:52:44.083776 (+ 24098us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.084487 (+ 711us) write_op.cc:454] Released schema lock
0905 19:52:44.459056 (+374569us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.459065 (+ 9us) write_op.cc:454] Released schema lock
0905 19:52:44.643032 (+183967us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.764703 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:43.764822 (+ 119us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:43.764830 (+ 8us) write_op.cc:435] Acquired schema lock
0905 19:52:43.764834 (+ 4us) tablet.cc:660] Decoding operations
0905 19:52:43.765095 (+ 261us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:43.765110 (+ 15us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:43.765115 (+ 5us) tablet.cc:683] Acquiring locks for 9 operations
0905 19:52:43.765305 (+ 190us) tablet.cc:699] Row locks acquired
0905 19:52:43.765308 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:43.765347 (+ 39us) write_op.cc:270] Start()
0905 19:52:43.765384 (+ 37us) write_op.cc:276] Timestamp: P: 1757101963765329 usec, L: 0
0905 19:52:43.765390 (+ 6us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.765601 (+ 211us) log.cc:844] Serialized 3520 byte log entry
0905 19:52:43.770162 (+ 4561us) op_driver.cc:464] REPLICATION: finished
0905 19:52:43.770311 (+ 149us) write_op.cc:301] APPLY: starting
0905 19:52:43.770327 (+ 16us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:43.770411 (+ 84us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:43.770467 (+ 56us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:43.770669 (+ 202us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:43.770708 (+ 39us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=9
0905 19:52:43.770715 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:43.770838 (+ 123us) log.cc:844] Serialized 71 byte log entry
0905 19:52:43.770901 (+ 63us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:43.770938 (+ 37us) write_op.cc:454] Released schema lock
0905 19:52:43.770945 (+ 7us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:44.042858 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:44.043005 (+ 147us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:44.043012 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:44.043014 (+ 2us) tablet.cc:660] Decoding operations
0905 19:52:44.043134 (+ 120us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:44.043153 (+ 19us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:44.043162 (+ 9us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:44.043242 (+ 80us) tablet.cc:699] Row locks acquired
0905 19:52:44.043245 (+ 3us) write_op.cc:260] PREPARE: finished
0905 19:52:44.043290 (+ 45us) write_op.cc:270] Start()
0905 19:52:44.043348 (+ 58us) write_op.cc:276] Timestamp: P: 1757101964043268 usec, L: 0
0905 19:52:44.043351 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:44.043551 (+ 200us) log.cc:844] Serialized 356 byte log entry
0905 19:52:44.048189 (+ 4638us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.048342 (+ 153us) write_op.cc:301] APPLY: starting
0905 19:52:44.048360 (+ 18us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:44.048405 (+ 45us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:44.048440 (+ 35us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:44.048503 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:44.048548 (+ 45us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:44.048556 (+ 8us) write_op.cc:312] APPLY: finished
0905 19:52:44.048638 (+ 82us) log.cc:844] Serialized 23 byte log entry
0905 19:52:44.048721 (+ 83us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.048740 (+ 19us) write_op.cc:454] Released schema lock
0905 19:52:44.048746 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:44.067268 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:44.067393 (+ 125us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:44.067399 (+ 6us) write_op.cc:435] Acquired schema lock
0905 19:52:44.067402 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:44.067513 (+ 111us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:44.067530 (+ 17us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:44.067535 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:44.067594 (+ 59us) tablet.cc:699] Row locks acquired
0905 19:52:44.067596 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:44.067653 (+ 57us) write_op.cc:270] Start()
0905 19:52:44.067688 (+ 35us) write_op.cc:276] Timestamp: P: 1757101964067613 usec, L: 0
0905 19:52:44.067693 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:44.067912 (+ 219us) log.cc:844] Serialized 356 byte log entry
0905 19:52:44.072435 (+ 4523us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.072592 (+ 157us) write_op.cc:301] APPLY: starting
0905 19:52:44.072609 (+ 17us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:44.072655 (+ 46us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:44.072683 (+ 28us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:44.072746 (+ 63us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:44.072786 (+ 40us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:44.072793 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:44.072881 (+ 88us) log.cc:844] Serialized 23 byte log entry
0905 19:52:44.072956 (+ 75us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.072981 (+ 25us) write_op.cc:454] Released schema lock
0905 19:52:44.072987 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Related trace 'op':
0905 19:52:44.400083 (+ 0us) write_op.cc:183] PREPARE: starting on tablet 00000000000000000000000000000000
0905 19:52:44.400214 (+ 131us) write_op.cc:432] Acquiring schema lock in shared mode
0905 19:52:44.400221 (+ 7us) write_op.cc:435] Acquired schema lock
0905 19:52:44.400224 (+ 3us) tablet.cc:660] Decoding operations
0905 19:52:44.400337 (+ 113us) write_op.cc:620] Acquiring the partition lock for write op
0905 19:52:44.400353 (+ 16us) write_op.cc:641] Partition lock acquired for write op
0905 19:52:44.400358 (+ 5us) tablet.cc:683] Acquiring locks for 1 operations
0905 19:52:44.400427 (+ 69us) tablet.cc:699] Row locks acquired
0905 19:52:44.400429 (+ 2us) write_op.cc:260] PREPARE: finished
0905 19:52:44.400468 (+ 39us) write_op.cc:270] Start()
0905 19:52:44.400504 (+ 36us) write_op.cc:276] Timestamp: P: 1757101964400395 usec, L: 0
0905 19:52:44.400509 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:44.400724 (+ 215us) log.cc:844] Serialized 356 byte log entry
0905 19:52:44.410246 (+ 9522us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.410398 (+ 152us) write_op.cc:301] APPLY: starting
0905 19:52:44.410413 (+ 15us) tablet.cc:1365] starting BulkCheckPresence
0905 19:52:44.410456 (+ 43us) tablet.cc:1368] finished BulkCheckPresence
0905 19:52:44.410515 (+ 59us) tablet.cc:1370] starting ApplyRowOperation cycle
0905 19:52:44.410605 (+ 90us) tablet.cc:1381] finished ApplyRowOperation cycle
0905 19:52:44.410638 (+ 33us) tablet_metrics.cc:563] ProbeStats: bloom_lookups=0,key_file_lookups=0,delta_file_lookups=0,mrs_lookups=1
0905 19:52:44.410645 (+ 7us) write_op.cc:312] APPLY: finished
0905 19:52:44.410768 (+ 123us) log.cc:844] Serialized 23 byte log entry
0905 19:52:44.410860 (+ 92us) write_op.cc:489] Releasing partition, row and schema locks
0905 19:52:44.410877 (+ 17us) write_op.cc:454] Released schema lock
0905 19:52:44.410883 (+ 6us) write_op.cc:341] FINISH: Updating metrics
Metrics: {"reported_tablets":161,"spinlock_wait_cycles":276117376,"child_traces":[["op",{"apply.queue_time_us":81,"apply.run_cpu_time_us":696,"apply.run_wall_time_us":695,"num_ops":9,"prepare.queue_time_us":47,"prepare.run_cpu_time_us":1018,"prepare.run_wall_time_us":1018,"raft.queue_time_us":945,"raft.run_cpu_time_us":263,"raft.run_wall_time_us":264,"replication_time_us":4741}],["op",{"apply.queue_time_us":96,"apply.run_cpu_time_us":460,"apply.run_wall_time_us":460,"num_ops":1,"prepare.queue_time_us":139,"prepare.run_cpu_time_us":807,"prepare.run_wall_time_us":807,"raft.queue_time_us":871,"raft.run_cpu_time_us":253,"raft.run_wall_time_us":253,"replication_time_us":4794}],["op",{"apply.queue_time_us":98,"apply.run_cpu_time_us":455,"apply.run_wall_time_us":454,"num_ops":1,"prepare.queue_time_us":179,"prepare.run_cpu_time_us":824,"prepare.run_wall_time_us":824,"raft.queue_time_us":170,"raft.run_cpu_time_us":248,"raft.run_wall_time_us":247,"replication_time_us":4705}],["op",{"apply.queue_time_us":88,"apply.run_cpu_time_us":540,"apply.run_wall_time_us":540,"num_ops":1,"prepare.queue_time_us":29187,"prepare.run_cpu_time_us":766,"prepare.run_wall_time_us":766,"raft.queue_time_us":1032,"raft.run_cpu_time_us":285,"raft.run_wall_time_us":286,"replication_time_us":9707}]]}
W20250905 19:52:44.652457 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 141299us
I20250905 19:52:44.653964 3749 tablet.cc:1722] T ae6fe4620dda4b1296416e01f8ffee29 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.657299 2397 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_6212ca6f887b4190bdf0b6f455c0bc3f [id=622da4d6f7234abd89d8917612a5d66e] alter complete (version 1)
I20250905 19:52:44.654846 3557 raft_consensus.cc:357] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.658653 2392 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 415508e72ee34bc391c4369b5f8a2f48
I20250905 19:52:44.658349 3557 raft_consensus.cc:383] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.659278 3749 tablet.cc:1722] T e10b4e82c4bc429b9a42db39c03a5267 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.659466 2392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_0f02f18fe4354cd193ba8216fd99d651"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:44.659184 3557 raft_consensus.cc:738] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
W20250905 19:52:44.658142 3815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20250905 19:52:44.660123 2392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_0f02f18fe4354cd193ba8216fd99d651 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.662951 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_e9c7c2eda1bf4355934378e90df10ea5"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:44.661950 3557 consensus_queue.cc:260] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:44.663524 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_e9c7c2eda1bf4355934378e90df10ea5 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.664777 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 415508e72ee34bc391c4369b5f8a2f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:44 UTC) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.666363 3539 log.cc:1199] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/d59da39e7bcf497a93fb9174328c1814
I20250905 19:52:44.666826 3539 ts_tablet_manager.cc:1939] T d59da39e7bcf497a93fb9174328c1814 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:44.666642 3557 ts_tablet_manager.cc:1428] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.023s user 0.004s sys 0.000s
I20250905 19:52:44.667510 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet d59da39e7bcf497a93fb9174328c1814 (table default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]) successfully deleted
I20250905 19:52:44.667901 3539 tablet_replica.cc:331] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:44.530320 744 tablet_service.cc:1468] Processing CreateTablet for tablet f6ac6052745e4524a0fad5d1c99dc064 (DEFAULT_TABLE table=default.Table_3dafac26db484c73a9482d5732d1fbf4 [id=51d1ade5c78342bfbb935326a74fe2bf]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.544714 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f439e2a456054e0b8c6fe10a76e7b7f1. 1 dirs total, 0 dirs full, 0 dirs failed
W20250905 19:52:44.669090 2403 catalog_manager.cc:5842] Failed to send alter table request: Not found: failed to reset TS proxy: No leader found: Tablet ed02bf5743f246a1937e9a9fe22c764e consensus state has no leader
I20250905 19:52:44.669631 2403 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_a97389de256643e9bb7d524b25747a4a"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.670177 2403 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_a97389de256643e9bb7d524b25747a4a in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.569391 712 leader_election.cc:304] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.671993 3677 tablet.cc:1722] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
W20250905 19:52:44.672117 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 102773us
I20250905 19:52:44.672600 3546 raft_consensus.cc:2802] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.597045 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98db7e2a9d6d4f4d984388a493295d72" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:44.672700 3546 raft_consensus.cc:491] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.672781 3546 raft_consensus.cc:3058] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.672797 764 raft_consensus.cc:3058] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.672576 712 leader_election.cc:304] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.675485 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f6ac6052745e4524a0fad5d1c99dc064. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.677368 3546 raft_consensus.cc:513] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.602869 743 tablet_service.cc:1515] Processing DeleteTablet for tablet 252afae7396d4d90b4647f4c4cc7b7cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:44 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.634927 1170 ts_tablet_manager.cc:1428] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.162s user 0.007s sys 0.012s
I20250905 19:52:44.662254 739 tablet_service.cc:1515] Processing DeleteTablet for tablet 415508e72ee34bc391c4369b5f8a2f48 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:44 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.515475 749 tablet_service.cc:1515] Processing DeleteTablet for tablet 160a7814abf9423dbe965c252c20fa7f with delete_type TABLET_DATA_DELETED (replaced by tablet 734937ccc1cf4a528247cc79f893e5ae) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.677795 764 raft_consensus.cc:2466] T 98db7e2a9d6d4f4d984388a493295d72 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:44.680296 848 leader_election.cc:304] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.680580 3681 raft_consensus.cc:2802] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:44.680699 3681 raft_consensus.cc:695] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Becoming Leader. State: Replica: 835daf30e4324782848290be181ae78f, State: Running, Role: LEADER
I20250905 19:52:44.680951 3681 consensus_queue.cc:237] T 98db7e2a9d6d4f4d984388a493295d72 P 835daf30e4324782848290be181ae78f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.679507 1170 tablet_bootstrap.cc:492] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.597999 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 74eb2632a6414e52b9ad67f53c182679 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-09-05 19:52:44 UTC) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.685568 1170 tablet_bootstrap.cc:654] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.685873 3802 raft_consensus.cc:2802] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:44.685992 3802 raft_consensus.cc:695] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
W20250905 19:52:44.686192 962 tablet.cc:2378] T 1110897f9ec14741b9b882b6e09081ab P 835daf30e4324782848290be181ae78f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:44.686203 3802 consensus_queue.cc:237] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.689136 2397 catalog_manager.cc:6127] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: default.Table_1f5a53ad37be48b1a6cea6f4da5f0e20 [id=3d53333f33bc4acd8572699247ce0e77] alter complete (version 1)
I20250905 19:52:44.689440 3546 leader_election.cc:290] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.698000 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_f6f01961676442768b28d13aa08504a6"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.699175 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_f6f01961676442768b28d13aa08504a6 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.700053 3557 tablet_bootstrap.cc:492] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.705020 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_41021d8a6ee248a7b3a9aaf30e726051"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.705611 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_41021d8a6ee248a7b3a9aaf30e726051 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:44.705679 986 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 52262.1us
I20250905 19:52:44.707923 2392 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_8b25ee22751f46379272b57257f5880d"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.708595 2392 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_8b25ee22751f46379272b57257f5880d in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.712677 2403 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_aef5a3f71caf4f50858bccf0a8494dde"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20250905 19:52:44.713069 3557 tablet_bootstrap.cc:654] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:44.713557 2403 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_aef5a3f71caf4f50858bccf0a8494dde in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20250905 19:52:44.714710 3817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:44.721891 3747 tablet.cc:1722] T 03cf686811e041e7a001669b3d86e28a P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:44.722071 1170 tablet_bootstrap.cc:492] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.722239 1170 ts_tablet_manager.cc:1397] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.043s user 0.000s sys 0.007s
I20250905 19:52:44.723170 1170 raft_consensus.cc:357] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.723403 1170 raft_consensus.cc:383] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.723471 1170 raft_consensus.cc:738] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.723737 1170 consensus_queue.cc:260] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.724347 1170 ts_tablet_manager.cc:1428] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20250905 19:52:44.724632 1170 tablet_bootstrap.cc:492] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
W20250905 19:52:44.726389 712 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.2:35993, user_credentials={real_user=slave}} blocked reactor thread for 53842.5us
I20250905 19:52:44.727137 3541 ts_tablet_manager.cc:1918] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.727136 712 leader_election.cc:304] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.727260 3541 log.cc:1199] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/6ee9f4fe3a464abaaca48f22ea183c0e
I20250905 19:52:44.727806 3541 ts_tablet_manager.cc:1939] T 6ee9f4fe3a464abaaca48f22ea183c0e P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.728610 3802 raft_consensus.cc:2802] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:44.728775 3802 raft_consensus.cc:695] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:44.729013 3802 consensus_queue.cc:237] T fe00969eb9654976a0efeac2fa7c91d3 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:44.731649 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 207933us
W20250905 19:52:44.732290 984 consensus_peers.cc:489] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: dc5428f20ca14fdf88a3a52f7b4d1e19. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.732792 3299 consensus_queue.cc:1035] T 298a325f5e4645beac39eb59ddc0566d P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.733455 3299 consensus_queue.cc:1035] T e77bdf9df6eb4470a3d22e96da952718 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.733526 3557 tablet_bootstrap.cc:492] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.733588 3540 server_base.cc:1047] running on GCE node
I20250905 19:52:44.733697 3557 ts_tablet_manager.cc:1397] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.034s user 0.003s sys 0.000s
I20250905 19:52:44.734072 984 leader_election.cc:304] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 835daf30e4324782848290be181ae78f
I20250905 19:52:44.734413 3820 tablet.cc:1722] T 3315d3e9633e42ea8d7a6b5934cb0438 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2
I20250905 19:52:44.734539 3557 raft_consensus.cc:357] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.734722 3557 raft_consensus.cc:383] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.734782 3557 raft_consensus.cc:738] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.734985 3557 consensus_queue.cc:260] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.735455 3557 ts_tablet_manager.cc:1428] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:44.735725 3557 tablet_bootstrap.cc:492] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.737428 3557 tablet_bootstrap.cc:654] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.744339 1170 tablet_bootstrap.cc:654] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:44.745334 848 consensus_peers.cc:489] T af5748bea3bd4f51a0d6c28a1d9044a0 P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: af5748bea3bd4f51a0d6c28a1d9044a0. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:44.747068 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 37) took 2460 ms. Trace:
I20250905 19:52:44.747180 3541 rpcz_store.cc:276] 0905 19:52:42.286961 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.287017 (+ 56us) service_pool.cc:225] Handling call
0905 19:52:44.595799 (+2308782us) ts_tablet_manager.cc:1145] Deleting tablet 6ee9f4fe3a464abaaca48f22ea183c0e
0905 19:52:44.595809 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.712662 (+116853us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.727022 (+ 14360us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.747041 (+ 20019us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2308589}
I20250905 19:52:44.747552 3541 tablet_replica.cc:331] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.747830 3541 raft_consensus.cc:2241] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.748034 3541 raft_consensus.cc:2270] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.748648 3541 ts_tablet_manager.cc:1905] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.750839 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 6ee9f4fe3a464abaaca48f22ea183c0e (table default.Table_264ad09719304e68b1ef82f9ec5e5c22 [id=15b5a51d6b824080b978428889335d39]) successfully deleted
I20250905 19:52:44.759173 3557 tablet_bootstrap.cc:492] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.759330 3557 ts_tablet_manager.cc:1397] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.004s sys 0.000s
I20250905 19:52:44.760304 3557 raft_consensus.cc:357] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.760471 3557 raft_consensus.cc:383] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.760519 3557 raft_consensus.cc:738] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.761229 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:44.761387 913 raft_consensus.cc:3058] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:44.761582 3828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20250905 19:52:44.763365 3540 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20250905 19:52:44.736054 1053 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "cea5667d882b4b57a6fa4e9866641bf6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:44.768322 913 raft_consensus.cc:2466] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:44.768832 3540 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20250905 19:52:44.768918 3610 raft_consensus.cc:2802] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.769048 3610 raft_consensus.cc:491] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.769140 3610 raft_consensus.cc:3058] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.769322 712 leader_election.cc:304] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.769670 3830 raft_consensus.cc:2802] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:44.769837 3830 raft_consensus.cc:695] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:44.770777 3830 consensus_queue.cc:237] T 7d2e58cc18b94f9ab036e3376920351c P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.773108 3610 raft_consensus.cc:513] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.774634 3539 raft_consensus.cc:2241] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.774914 3539 raft_consensus.cc:2270] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.776010 3820 tablet.cc:1722] T 3d3a7b03d5e34f8dbcd5b6b994c9b419 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.778680 3557 consensus_queue.cc:260] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.779181 3557 ts_tablet_manager.cc:1428] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.020s user 0.002s sys 0.000s
I20250905 19:52:44.781793 1170 tablet_bootstrap.cc:492] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.781932 1170 ts_tablet_manager.cc:1397] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.057s user 0.003s sys 0.000s
I20250905 19:52:44.782718 1170 raft_consensus.cc:357] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.782882 1170 raft_consensus.cc:383] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.782936 1170 raft_consensus.cc:738] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.783114 1170 consensus_queue.cc:260] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.783608 1170 ts_tablet_manager.cc:1428] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20250905 19:52:44.783881 1170 tablet_bootstrap.cc:492] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.784193 3540 hybrid_clock.cc:648] HybridClock initialized: now 1757101964783996 us; error 198 us; skew 500 ppm
I20250905 19:52:44.784955 3540 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20250905 19:52:44.785351 1170 tablet_bootstrap.cc:654] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.779475 3557 tablet_bootstrap.cc:492] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.786532 2373 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_371a560776cb4bf1bca8080a67cd445e"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.787317 2373 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_371a560776cb4bf1bca8080a67cd445e in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.787863 3557 tablet_bootstrap.cc:654] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.791077 3541 ts_tablet_manager.cc:1918] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.791175 3541 log.cc:1199] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/d7155cf5abba4e2cbdce583406f94fad
I20250905 19:52:44.791483 3541 ts_tablet_manager.cc:1939] T d7155cf5abba4e2cbdce583406f94fad P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.794217 1170 tablet_bootstrap.cc:492] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.794350 1170 ts_tablet_manager.cc:1397] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.000s
I20250905 19:52:44.795038 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 38) took 2507 ms. Trace:
I20250905 19:52:44.795038 1170 raft_consensus.cc:357] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.795248 1170 raft_consensus.cc:383] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.795305 1170 raft_consensus.cc:738] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.795120 3541 rpcz_store.cc:276] 0905 19:52:42.287340 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.287380 (+ 40us) service_pool.cc:225] Handling call
0905 19:52:44.747502 (+2460122us) ts_tablet_manager.cc:1145] Deleting tablet d7155cf5abba4e2cbdce583406f94fad
0905 19:52:44.747512 (+ 10us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.777040 (+ 29528us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.790962 (+ 13922us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.795014 (+ 4052us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2459971}
I20250905 19:52:44.795537 1170 consensus_queue.cc:260] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.795696 3541 tablet_replica.cc:331] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.795871 3541 raft_consensus.cc:2241] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:44.796064 1170 ts_tablet_manager.cc:1428] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:44.796157 3541 raft_consensus.cc:2270] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.796526 1170 tablet_bootstrap.cc:492] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.796689 3541 ts_tablet_manager.cc:1905] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.797405 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet d7155cf5abba4e2cbdce583406f94fad (table default.Table_34a19942c4604178a622037fc5303a1c [id=eed85872c9304732a9dfaa50c65ff816]) successfully deleted
I20250905 19:52:44.797864 1170 tablet_bootstrap.cc:654] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.800246 2370 catalog_manager.cc:2232] Servicing CreateTable request from {username='slave'} at 127.0.0.1:34736:
name: "default.Table_539cc5e7803143249b4335723984ab48"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20250905 19:52:44.800869 2370 catalog_manager.cc:6944] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table default.Table_539cc5e7803143249b4335723984ab48 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20250905 19:52:44.802520 2403 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6e40598f9dbc40ac9860eb5b78130a93" } modify_external_catalogs: true
I20250905 19:52:44.802739 2403 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6e40598f9dbc40ac9860eb5b78130a93" } modify_external_catalogs: true
I20250905 19:52:44.804215 2405 catalog_manager.cc:5869] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711: Sending DeleteTablet for 3 replicas of tablet 0fb8fe93061848a0b055979bad10f08b
I20250905 19:52:44.804747 3540 webserver.cc:480] Webserver started at http://127.0.104.60:44671/ using document root <none> and password file <none>
I20250905 19:52:44.804834 2405 catalog_manager.cc:6652] ReplaceTablet: tablet 0fb8fe93061848a0b055979bad10f08b deleted and replaced by tablet f9c45ce5d9dc4bab96f95e2fd6495698
I20250905 19:52:44.804865 3557 tablet_bootstrap.cc:492] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.804986 3557 ts_tablet_manager.cc:1397] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.003s sys 0.000s
I20250905 19:52:44.805939 3540 fs_manager.cc:362] Metadata directory not provided
I20250905 19:52:44.806080 3540 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20250905 19:52:44.806025 3557 raft_consensus.cc:357] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.806188 3557 raft_consensus.cc:383] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.806244 3557 raft_consensus.cc:738] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.806584 3557 consensus_queue.cc:260] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.808123 3557 ts_tablet_manager.cc:1428] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20250905 19:52:44.808200 2405 catalog_manager.cc:2482] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_0d6fe65405f543d7a70643f94bf22da5" } modify_external_catalogs: true
I20250905 19:52:44.808389 2405 catalog_manager.cc:2730] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_0d6fe65405f543d7a70643f94bf22da5" } modify_external_catalogs: true
I20250905 19:52:44.809207 3557 tablet_bootstrap.cc:492] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.811409 3557 tablet_bootstrap.cc:654] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.813899 2392 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_858dac3c36484123a1a9de7c20d0f141" } new_table_name: "default.Table_d14d94b8d93e4b79a8a8da42ab581478" modify_external_catalogs: true
I20250905 19:52:44.818807 3830 consensus_queue.cc:1035] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:44.819448 752 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fb8fe93061848a0b055979bad10f08b with delete_type TABLET_DATA_DELETED (replaced by tablet f9c45ce5d9dc4bab96f95e2fd6495698) from {username='slave'} at 127.0.0.1:37950
I20250905 19:52:44.820298 877 tablet_service.cc:1515] Processing DeleteTablet for tablet 0fb8fe93061848a0b055979bad10f08b with delete_type TABLET_DATA_DELETED (replaced by tablet f9c45ce5d9dc4bab96f95e2fd6495698) from {username='slave'} at 127.0.0.1:60504
I20250905 19:52:44.821769 3557 tablet_bootstrap.cc:492] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:44.821907 3557 ts_tablet_manager.cc:1397] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.000s
I20250905 19:52:44.827678 2373 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_6e9009b9124c413eb5058a1f3294990a" } new_table_name: "default.Table_a9c2350d17804324be594a0419a28cbe" modify_external_catalogs: true
I20250905 19:52:44.827680 3557 raft_consensus.cc:357] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.827896 3557 raft_consensus.cc:383] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.828013 3557 raft_consensus.cc:738] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.828261 3557 consensus_queue.cc:260] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.829488 3541 ts_tablet_manager.cc:1918] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:44.829622 3541 log.cc:1199] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/ad0760dfee934aacbd4ab1fb81e116c7
I20250905 19:52:44.829674 3557 ts_tablet_manager.cc:1428] T ed8dae95b500437d869548995d84451f P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20250905 19:52:44.830156 3557 tablet_bootstrap.cc:492] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:44.830569 1170 tablet_bootstrap.cc:492] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:44.830724 1170 ts_tablet_manager.cc:1397] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.034s user 0.004s sys 0.002s
I20250905 19:52:44.831452 1170 raft_consensus.cc:357] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.831617 1170 raft_consensus.cc:383] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.831691 1170 raft_consensus.cc:738] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.831955 3557 tablet_bootstrap.cc:654] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.831892 1170 consensus_queue.cc:260] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.832262 3541 ts_tablet_manager.cc:1939] T ad0760dfee934aacbd4ab1fb81e116c7 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.833206 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 39) took 2448 ms. Trace:
I20250905 19:52:44.833305 3541 rpcz_store.cc:276] 0905 19:52:42.385140 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.386294 (+ 1154us) service_pool.cc:225] Handling call
0905 19:52:44.795632 (+2409338us) ts_tablet_manager.cc:1145] Deleting tablet ad0760dfee934aacbd4ab1fb81e116c7
0905 19:52:44.795643 (+ 11us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.799820 (+ 4177us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.829382 (+ 29562us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.833158 (+ 3776us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2409114}
I20250905 19:52:44.833750 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet ad0760dfee934aacbd4ab1fb81e116c7 (table default.Table_44c46e2e89984ccfabdeee5a59adbf7b [id=8e11def15ad743419ab4c2fea50888cf]) successfully deleted
I20250905 19:52:44.834286 3541 tablet_replica.cc:331] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.834494 3541 raft_consensus.cc:2241] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:44.840837 1170 ts_tablet_manager.cc:1428] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20250905 19:52:44.841176 1170 tablet_bootstrap.cc:492] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe: Bootstrap starting.
I20250905 19:52:44.842406 3541 raft_consensus.cc:2270] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.843518 3541 ts_tablet_manager.cc:1905] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.847308 1170 tablet_bootstrap.cc:654] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:44.739087 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2edf591963ca46369c29a3fb2bf24bc3" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:44.740249 1169 tablet_bootstrap.cc:492] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:44.853401 1169 ts_tablet_manager.cc:1397] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.553s user 0.012s sys 0.001s
I20250905 19:52:44.743068 3834 tablet.cc:1722] T fbba1ad28a614b06afe865c478901a32 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.853751 3539 ts_tablet_manager.cc:1905] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.858994 3834 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:50082 (request call id 76) took 1175 ms. Trace:
I20250905 19:52:44.859141 3834 rpcz_store.cc:276] 0905 19:52:43.683261 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.683325 (+ 64us) service_pool.cc:225] Handling call
0905 19:52:44.858980 (+1175655us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.683695 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:43.683787 (+ 92us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:43.683796 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:43.683803 (+ 7us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:43.683832 (+ 29us) alter_schema_op.cc:128] START. Timestamp: 1757101963683661
0905 19:52:43.683835 (+ 3us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.684009 (+ 174us) log.cc:844] Serialized 173 byte log entry
0905 19:52:44.742605 (+1058596us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.743051 (+ 446us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:44.853848 (+110797us) log.cc:844] Serialized 5 byte log entry
0905 19:52:44.857590 (+ 3742us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.858519 (+ 929us) log.cc:844] Serialized 14 byte log entry
0905 19:52:44.858943 (+ 424us) alter_schema_op.cc:73] Released schema lock
0905 19:52:44.858949 (+ 6us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":373,"prepare.queue_time_us":55,"prepare.run_cpu_time_us":67553,"prepare.run_wall_time_us":280389,"raft.queue_time_us":31,"raft.run_cpu_time_us":306,"raft.run_wall_time_us":305,"replication_time_us":1058729,"spinlock_wait_cycles":16256,"thread_start_us":280226,"threads_started":2,"wal-append.queue_time_us":279005}]]}
I20250905 19:52:44.860502 3834 tablet.cc:1722] T d495b6ba92e84b588b4780c2896195c0 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.860998 3835 tablet.cc:1722] T 414f30a772524c33a74864ff741c865f P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.740896 3749 tablet.cc:1722] T f034b8cb5d8943938cb2b58c33b45494 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.738878 3547 ts_tablet_manager.cc:1918] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:44.865028 3547 log.cc:1199] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-2/wal/wals/e58bbb6575fd4437b242032a281914ea
I20250905 19:52:44.864993 3791 raft_consensus.cc:491] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.782923 3747 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:50082 (request call id 78) took 1099 ms. Trace:
I20250905 19:52:44.865173 3791 raft_consensus.cc:513] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.865828 3791 leader_election.cc:290] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.866921 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97912167c32646d18f005a9039d6eca6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:44.867425 848 leader_election.cc:343] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.868723 3547 ts_tablet_manager.cc:1939] T e58bbb6575fd4437b242032a281914ea P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting consensus metadata
I20250905 19:52:44.868986 3541 ts_tablet_manager.cc:1918] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:44.869148 3541 log.cc:1199] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/b3ea2e2059a4425892fa3c0757e69e4c
I20250905 19:52:44.869589 3541 ts_tablet_manager.cc:1939] T b3ea2e2059a4425892fa3c0757e69e4c P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:44.869987 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 40) took 2302 ms. Trace:
I20250905 19:52:44.870110 3541 rpcz_store.cc:276] 0905 19:52:42.567710 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.567787 (+ 77us) service_pool.cc:225] Handling call
0905 19:52:44.834212 (+2266425us) ts_tablet_manager.cc:1145] Deleting tablet b3ea2e2059a4425892fa3c0757e69e4c
0905 19:52:44.834221 (+ 9us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.859195 (+ 24974us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.868828 (+ 9633us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.869940 (+ 1112us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2266181}
I20250905 19:52:44.870654 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet b3ea2e2059a4425892fa3c0757e69e4c (table default.Table_6e52652f409647a88e809923474b09dd [id=3e181278985a4082a8173307e0a028d0]) successfully deleted
I20250905 19:52:44.865319 3747 rpcz_store.cc:276] 0905 19:52:43.683919 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.683993 (+ 74us) service_pool.cc:225] Handling call
0905 19:52:44.782903 (+1098910us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.745806 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:43.745966 (+ 160us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:43.745979 (+ 13us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:43.745990 (+ 11us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:43.746059 (+ 69us) alter_schema_op.cc:128] START. Timestamp: 1757101963745875
0905 19:52:43.746064 (+ 5us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.746325 (+ 261us) log.cc:844] Serialized 173 byte log entry
0905 19:52:44.707080 (+960755us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.721850 (+ 14770us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:44.722630 (+ 780us) log.cc:844] Serialized 5 byte log entry
0905 19:52:44.727291 (+ 4661us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.782084 (+ 54793us) log.cc:844] Serialized 14 byte log entry
0905 19:52:44.782864 (+ 780us) alter_schema_op.cc:73] Released schema lock
0905 19:52:44.782869 (+ 5us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":14680,"prepare.queue_time_us":61521,"prepare.run_cpu_time_us":894,"prepare.run_wall_time_us":1652,"raft.queue_time_us":38,"raft.run_cpu_time_us":304,"raft.run_wall_time_us":304,"replication_time_us":960981,"thread_start_us":958,"threads_started":1,"wal-append.queue_time_us":461}]]}
I20250905 19:52:44.872500 3747 tablet.cc:1722] T f6f29ac5e68d4d8da8f3496fefe695ec P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.876230 3677 tablet.cc:1722] T af10abf0a5b34b3bbfea20e5c2005006 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.885944 3677 tablet.cc:1722] T a860ab3f5ce54bd2b75bb71730a3c6e6 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:44.891280 900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:44.891541 900 raft_consensus.cc:2391] T 7d2e58cc18b94f9ab036e3376920351c P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in current term 1: Already voted for candidate f21c498a0d964e8fbebfd1b3cb1916fe in this term.
I20250905 19:52:44.895641 900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:44.895860 900 raft_consensus.cc:3058] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.896822 3837 raft_consensus.cc:491] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.896960 3837 raft_consensus.cc:513] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.897187 3541 tablet_replica.cc:331] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.897488 3541 raft_consensus.cc:2241] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:44.897728 3541 raft_consensus.cc:2270] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.897938 3837 leader_election.cc:290] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.898414 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a95130c82e14bbcae577d4244b876ab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.898628 902 raft_consensus.cc:2466] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:44.899055 712 leader_election.cc:304] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.899369 3837 raft_consensus.cc:2802] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:44.899528 3837 raft_consensus.cc:491] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:44.899691 3837 raft_consensus.cc:3058] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.900971 900 raft_consensus.cc:2466] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
I20250905 19:52:44.903421 3837 raft_consensus.cc:513] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.904524 3837 leader_election.cc:290] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.904760 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a95130c82e14bbcae577d4244b876ab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:44.904974 902 raft_consensus.cc:3058] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:44.906437 3541 ts_tablet_manager.cc:1905] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.907923 902 raft_consensus.cc:2466] T 4a95130c82e14bbcae577d4244b876ab P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:44.908797 712 leader_election.cc:304] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:44.909188 3837 raft_consensus.cc:2802] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:44.909528 3837 raft_consensus.cc:695] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:44.910171 3837 consensus_queue.cc:237] T 4a95130c82e14bbcae577d4244b876ab P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
W20250905 19:52:44.852001 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 83669.8us
I20250905 19:52:44.923990 3837 raft_consensus.cc:491] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.924223 3837 raft_consensus.cc:513] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.925159 3837 leader_election.cc:290] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:44.925650 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a104b1c8ce14fa2bf101e1b34183cfa" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:44.926776 3677 tablet.cc:1722] T 85c648f799304fa9a94bec665a2dcf97 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 2 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 3
I20250905 19:52:44.853403 3610 leader_election.cc:290] T 1110897f9ec14741b9b882b6e09081ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Requested vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
W20250905 19:52:44.928982 984 consensus_peers.cc:489] T 4edfa9a656554e02b6649e8430c215fb P 4eb3af79b01d4835bf6bf7b26bf016f5 -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 4edfa9a656554e02b6649e8430c215fb. This is attempt 1: this message will repeat every 5th retry.
W20250905 19:52:44.935118 984 leader_election.cc:343] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.938969 984 leader_election.cc:304] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 4eb3af79b01d4835bf6bf7b26bf016f5; no voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:44.939759 822 heartbeater.cc:499] Master 127.0.104.62:38687 was elected leader, sending a full tablet report...
I20250905 19:52:44.929791 1169 raft_consensus.cc:357] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.940199 1169 raft_consensus.cc:383] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:44.940315 1169 raft_consensus.cc:738] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:44.940551 1169 consensus_queue.cc:260] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.941263 1169 ts_tablet_manager.cc:1428] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.077s user 0.000s sys 0.002s
I20250905 19:52:44.930138 3547 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:50082 (request call id 70) took 1871 ms. Trace:
I20250905 19:52:44.941468 3547 rpcz_store.cc:276] 0905 19:52:43.059048 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.060146 (+ 1098us) service_pool.cc:225] Handling call
0905 19:52:44.349445 (+1289299us) ts_tablet_manager.cc:1145] Deleting tablet e58bbb6575fd4437b242032a281914ea
0905 19:52:44.349533 (+ 88us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.454305 (+104772us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.738777 (+284472us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.930120 (+191343us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1289085}
I20250905 19:52:44.941931 3547 tablet_replica.cc:331] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: stopping tablet replica
I20250905 19:52:44.942149 3547 raft_consensus.cc:2241] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:44.942471 3547 raft_consensus.cc:2270] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.943114 3547 ts_tablet_manager.cc:1905] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 4eb3af79b01d4835bf6bf7b26bf016f5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.945027 3610 raft_consensus.cc:2747] T 30e1ca2bd1f94f748f2022b5669bc07e P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20250905 19:52:44.947825 2370 catalog_manager.cc:3502] Servicing AlterTable request from {username='slave'} at 127.0.0.1:34736:
table { table_name: "default.Table_4074e68bdbd84b8ab0734c59a600b1b7" } new_table_name: "default.Table_65a994935a794f7a98df2a1632ddcd3b" modify_external_catalogs: true
I20250905 19:52:44.950407 3791 raft_consensus.cc:491] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.950554 3791 raft_consensus.cc:513] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.951133 3791 leader_election.cc:290] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.952133 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "350197e134c549198771f4f8337d5dfc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:44.952798 848 leader_election.cc:343] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.953982 3541 ts_tablet_manager.cc:1918] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:44.954113 3541 log.cc:1199] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/4a9afd0fd4ad4ea0afcab9d7291a76bb
I20250905 19:52:44.955816 3541 ts_tablet_manager.cc:1939] T 4a9afd0fd4ad4ea0afcab9d7291a76bb P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
W20250905 19:52:44.957592 1101 tablet.cc:2378] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20250905 19:52:44.958271 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 53) took 2166 ms. Trace:
I20250905 19:52:44.958376 3541 rpcz_store.cc:276] 0905 19:52:42.791527 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:42.791595 (+ 68us) service_pool.cc:225] Handling call
0905 19:52:44.897108 (+2105513us) ts_tablet_manager.cc:1145] Deleting tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb
0905 19:52:44.897122 (+ 14us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.924716 (+ 27594us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.931128 (+ 6412us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:44.958229 (+ 27101us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":2005807}
I20250905 19:52:44.958791 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (table default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]) successfully deleted
I20250905 19:52:44.965096 3541 tablet_replica.cc:331] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:44.965422 3541 raft_consensus.cc:2241] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 1 LEADER]: Raft consensus shutting down.
I20250905 19:52:44.965794 3541 raft_consensus.cc:2270] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:44.966522 3541 ts_tablet_manager.cc:1905] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:44.979185 3681 raft_consensus.cc:491] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:44.979352 3681 raft_consensus.cc:513] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:44.979995 3681 leader_election.cc:290] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:44.980746 773 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca00bfe551a34ca9bb6d14b3a24596b6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:44.981257 848 leader_election.cc:343] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:44.992434 984 leader_election.cc:343] T e476304b492f4bd69d5ea0f109263517 P 4eb3af79b01d4835bf6bf7b26bf016f5 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:44.993992 752 tablet_service.cc:1468] Processing CreateTablet for tablet 810d8c3ac0834eebbaf23f5b01584563 (DEFAULT_TABLE table=default.Table_31d60118961a446dbd4a3e75b5afb0cc [id=d0b87591a5e74e23aedd00d97de0be0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.994441 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810d8c3ac0834eebbaf23f5b01584563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:44.997203 878 tablet_service.cc:1468] Processing CreateTablet for tablet 1adb31bada464449b6927f3dfd62b63a (DEFAULT_TABLE table=default.Table_a03f6fcd38264fce9c74b6e716c783af [id=5a72b522086645f99293033bdb1ef436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:44.997743 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1adb31bada464449b6927f3dfd62b63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.000186 877 tablet_service.cc:1468] Processing CreateTablet for tablet 810d8c3ac0834eebbaf23f5b01584563 (DEFAULT_TABLE table=default.Table_31d60118961a446dbd4a3e75b5afb0cc [id=d0b87591a5e74e23aedd00d97de0be0e]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.000659 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 810d8c3ac0834eebbaf23f5b01584563. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.004879 741 tablet_service.cc:1468] Processing CreateTablet for tablet 1adb31bada464449b6927f3dfd62b63a (DEFAULT_TABLE table=default.Table_a03f6fcd38264fce9c74b6e716c783af [id=5a72b522086645f99293033bdb1ef436]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.005362 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1adb31bada464449b6927f3dfd62b63a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.004879 749 tablet_service.cc:1468] Processing CreateTablet for tablet 6a10c1cfd18149b1afc933baf4654467 (DEFAULT_TABLE table=default.Table_a6c14ced6a2f4333985310637ee47d5f [id=1a7a2da030e740c6b3e1f3896b63cb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.008044 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a10c1cfd18149b1afc933baf4654467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.012324 3802 raft_consensus.cc:491] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.012450 3802 raft_consensus.cc:513] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.012863 3681 raft_consensus.cc:491] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.013010 3681 raft_consensus.cc:513] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.013059 3802 leader_election.cc:290] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.013182 3802 raft_consensus.cc:491] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.013264 3802 raft_consensus.cc:513] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.013547 3681 leader_election.cc:290] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:45.013648 3802 leader_election.cc:290] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.014930 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736bfce682e64498a8dde3310305536a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:45.016121 775 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26905425077e4746bccb51431da706c9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:45.016705 848 leader_election.cc:343] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.016835 913 raft_consensus.cc:2466] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.017143 712 leader_election.cc:304] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.018635 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ca3fc7055c44df282285806ea151e51" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:45.018813 902 raft_consensus.cc:2466] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.019129 712 leader_election.cc:304] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.020260 3802 raft_consensus.cc:2802] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:45.020367 3802 raft_consensus.cc:491] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:45.020424 3802 raft_consensus.cc:3058] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.022797 3802 raft_consensus.cc:513] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.024171 3802 leader_election.cc:290] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.024554 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736bfce682e64498a8dde3310305536a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:45.024731 902 raft_consensus.cc:3058] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.027536 902 raft_consensus.cc:2466] T 736bfce682e64498a8dde3310305536a P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:45.016544 3541 ts_tablet_manager.cc:1918] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:45.028618 3541 log.cc:1199] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/e58bbb6575fd4437b242032a281914ea
I20250905 19:52:45.000160 881 tablet_service.cc:1468] Processing CreateTablet for tablet 7cf94388773e4864981c62984530bbbf (DEFAULT_TABLE table=default.Table_28de77aaa9fd4ae2b3336689da11bda1 [id=d052c52370f9404e954f4882cc14e7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.029027 3541 ts_tablet_manager.cc:1939] T e58bbb6575fd4437b242032a281914ea P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:45.029425 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf94388773e4864981c62984530bbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.029749 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 70) took 1901 ms. Trace:
I20250905 19:52:45.029865 3541 rpcz_store.cc:276] 0905 19:52:43.128305 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.128356 (+ 51us) service_pool.cc:225] Handling call
0905 19:52:44.964730 (+1836374us) ts_tablet_manager.cc:1145] Deleting tablet e58bbb6575fd4437b242032a281914ea
0905 19:52:44.965007 (+ 277us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:44.990246 (+ 25239us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:45.016418 (+ 26172us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:45.029705 (+ 13287us) inbound_call.cc:173] Queueing success response
Metrics: {"tablet-delete.queue_time_us":1794399}
I20250905 19:52:45.030267 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]) successfully deleted
I20250905 19:52:45.036381 3151 consensus_queue.cc:1035] T 6b8bbbcac06944469e13f14ab1d80402 P 4eb3af79b01d4835bf6bf7b26bf016f5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:45.036587 3541 tablet_replica.cc:331] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:45.036898 3541 raft_consensus.cc:2241] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:45.037146 3541 raft_consensus.cc:2270] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:45.037891 3541 ts_tablet_manager.cc:1905] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:45.024510 744 tablet_service.cc:1468] Processing CreateTablet for tablet 7cf94388773e4864981c62984530bbbf (DEFAULT_TABLE table=default.Table_28de77aaa9fd4ae2b3336689da11bda1 [id=d052c52370f9404e954f4882cc14e7f9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.028358 712 leader_election.cc:304] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.038960 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7cf94388773e4864981c62984530bbbf. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.036815 3837 raft_consensus.cc:2802] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:45.041177 3837 raft_consensus.cc:491] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:45.041343 3837 raft_consensus.cc:3058] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.044064 3802 raft_consensus.cc:2802] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:45.044283 3802 raft_consensus.cc:695] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:45.044770 3802 consensus_queue.cc:237] T 736bfce682e64498a8dde3310305536a P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.046785 3837 raft_consensus.cc:513] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.047681 3837 leader_election.cc:290] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.048851 3834 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:50082 (request call id 79) took 1364 ms. Trace:
I20250905 19:52:45.048988 3834 rpcz_store.cc:276] 0905 19:52:43.684369 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.684409 (+ 40us) service_pool.cc:225] Handling call
0905 19:52:45.048827 (+1364418us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.747480 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:43.747607 (+ 127us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:43.747616 (+ 9us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:43.747622 (+ 6us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:43.747680 (+ 58us) alter_schema_op.cc:128] START. Timestamp: 1757101963747483
0905 19:52:43.747682 (+ 2us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.747819 (+ 137us) log.cc:844] Serialized 173 byte log entry
0905 19:52:44.743661 (+995842us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.860477 (+116816us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:44.861030 (+ 553us) log.cc:844] Serialized 5 byte log entry
0905 19:52:44.871046 (+ 10016us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:45.048525 (+177479us) log.cc:844] Serialized 14 byte log entry
0905 19:52:45.048781 (+ 256us) alter_schema_op.cc:73] Released schema lock
0905 19:52:45.048788 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":116555,"mutex_wait_us":115018,"prepare.queue_time_us":62845,"prepare.run_cpu_time_us":75712,"prepare.run_wall_time_us":276628,"raft.queue_time_us":78,"raft.run_cpu_time_us":392,"raft.run_wall_time_us":466,"replication_time_us":995942,"thread_start_us":504370,"threads_started":3,"wal-append.queue_time_us":274545}]]}
I20250905 19:52:45.049887 3834 tablet.cc:1722] T d011d1ec8f634386872e7ab257de3187 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.050050 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ca3fc7055c44df282285806ea151e51" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:45.050294 902 raft_consensus.cc:3058] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.050601 3802 raft_consensus.cc:491] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.050742 3802 raft_consensus.cc:513] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.051349 3802 leader_election.cc:290] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.051697 1169 tablet_bootstrap.cc:492] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:45.051733 913 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2edf591963ca46369c29a3fb2bf24bc3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:45.052102 913 raft_consensus.cc:2466] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.052433 712 leader_election.cc:304] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.053238 902 raft_consensus.cc:2466] T 9ca3fc7055c44df282285806ea151e51 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:45.053630 712 leader_election.cc:304] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.054370 1169 tablet_bootstrap.cc:654] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.055419 3802 raft_consensus.cc:2802] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:45.055562 3802 raft_consensus.cc:491] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:45.055724 3802 raft_consensus.cc:3058] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.058166 3801 raft_consensus.cc:2802] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:45.058363 3801 raft_consensus.cc:695] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:45.060698 3677 tablet.cc:1722] T ae6fe4620dda4b1296416e01f8ffee29 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.062217 3859 tablet.cc:1722] T e10b4e82c4bc429b9a42db39c03a5267 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.063854 3837 raft_consensus.cc:491] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.064064 3837 raft_consensus.cc:513] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.064970 3837 leader_election.cc:290] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.065397 3860 tablet.cc:1722] T 03cf686811e041e7a001669b3d86e28a P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 4
I20250905 19:52:45.067904 1169 tablet_bootstrap.cc:492] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5: No bootstrap required, opened a new log
I20250905 19:52:45.068177 1169 ts_tablet_manager.cc:1397] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.001s
I20250905 19:52:45.072348 1169 raft_consensus.cc:357] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.072757 1169 raft_consensus.cc:383] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.072870 1169 raft_consensus.cc:738] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4eb3af79b01d4835bf6bf7b26bf016f5, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.073163 1169 consensus_queue.cc:260] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.074780 1169 ts_tablet_manager.cc:1428] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20250905 19:52:45.074783 3802 raft_consensus.cc:513] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.076025 1169 tablet_bootstrap.cc:492] T 3c5cf2bc19b043f68838f8a6af6393cd P 4eb3af79b01d4835bf6bf7b26bf016f5: Bootstrap starting.
I20250905 19:52:45.076323 900 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c200681d4d77425389558e6489fb6032" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:45.070295 3835 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.AlterSchema from 127.0.0.1:50082 (request call id 77) took 1386 ms. Trace:
W20250905 19:52:45.070502 984 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 78087.7us
W20250905 19:52:45.070672 3749 tablet_replica.cc:1408] Time spent flushing the Log queue took a long time: real 0.206s user 0.004s sys 0.000s
I20250905 19:52:45.058686 3801 consensus_queue.cc:237] T 9ca3fc7055c44df282285806ea151e51 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.077708 3541 ts_tablet_manager.cc:1918] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:45.082270 3541 log.cc:1199] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/dc5428f20ca14fdf88a3a52f7b4d1e19
I20250905 19:52:45.082479 3859 tablet.cc:1722] T d495b6ba92e84b588b4780c2896195c0 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.082813 3541 ts_tablet_manager.cc:1939] T dc5428f20ca14fdf88a3a52f7b4d1e19 P 835daf30e4324782848290be181ae78f: Deleting consensus metadata
I20250905 19:52:45.082911 900 raft_consensus.cc:2466] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.083274 712 leader_election.cc:304] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.083357 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2edf591963ca46369c29a3fb2bf24bc3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:45.083567 902 raft_consensus.cc:3058] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.084110 3802 leader_election.cc:290] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.084273 3802 raft_consensus.cc:2802] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Leader pre-election won for term 1
I20250905 19:52:45.084334 3802 raft_consensus.cc:491] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20250905 19:52:45.084393 3802 raft_consensus.cc:3058] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.091297 3541 rpcz_store.cc:275] Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:60504 (request call id 71) took 1962 ms. Trace:
I20250905 19:52:45.092765 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "98db7e2a9d6d4f4d984388a493295d72" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.092952 1045 raft_consensus.cc:3058] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
W20250905 19:52:45.096750 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 181704us
I20250905 19:52:45.097743 3802 raft_consensus.cc:513] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.100390 3802 leader_election.cc:290] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Requested vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.095064 3677 tablet.cc:1722] T fbba1ad28a614b06afe865c478901a32 P 835daf30e4324782848290be181ae78f: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.097502 902 raft_consensus.cc:2466] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:45.101426 911 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c200681d4d77425389558e6489fb6032" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f"
I20250905 19:52:45.101783 3541 rpcz_store.cc:276] 0905 19:52:43.128397 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.131689 (+ 3292us) service_pool.cc:225] Handling call
0905 19:52:45.036484 (+1904795us) ts_tablet_manager.cc:1145] Deleting tablet dc5428f20ca14fdf88a3a52f7b4d1e19
0905 19:52:45.036501 (+ 17us) ts_tablet_manager.cc:1065] Acquired tablet manager lock
0905 19:52:45.057355 (+ 20854us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:45.077573 (+ 20218us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:45.091248 (+ 13675us) inbound_call.cc:173] Queueing success response
Metrics: {"mutex_wait_us":6685,"tablet-delete.queue_time_us":1858965}
I20250905 19:52:45.107283 911 raft_consensus.cc:3058] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.107440 3541 tablet_replica.cc:331] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: stopping tablet replica
I20250905 19:52:45.107445 2292 catalog_manager.cc:4928] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993): tablet dc5428f20ca14fdf88a3a52f7b4d1e19 (table default.Table_2c0cad91758c4851893e8a5919f5d44c [id=4cbf977f95a945daa1be1e19c78c4786]) successfully deleted
I20250905 19:52:45.107700 3541 raft_consensus.cc:2241] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:45.107926 3541 raft_consensus.cc:2270] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:45.108630 3541 ts_tablet_manager.cc:1905] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20250905 19:52:45.110152 3557 tablet_bootstrap.cc:492] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.110301 3557 ts_tablet_manager.cc:1397] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.280s user 0.060s sys 0.058s
I20250905 19:52:45.110450 877 tablet_service.cc:1468] Processing CreateTablet for tablet 6a10c1cfd18149b1afc933baf4654467 (DEFAULT_TABLE table=default.Table_a6c14ced6a2f4333985310637ee47d5f [id=1a7a2da030e740c6b3e1f3896b63cb0d]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.110631 911 raft_consensus.cc:2466] T c200681d4d77425389558e6489fb6032 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:45.111097 712 leader_election.cc:304] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.111200 3557 raft_consensus.cc:357] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.111379 3557 raft_consensus.cc:383] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.111456 3557 raft_consensus.cc:738] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.111613 3802 raft_consensus.cc:2802] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:45.111500 741 tablet_service.cc:1468] Processing CreateTablet for tablet e1bbc1342ffc4c0ab4508b8e6e30409f (DEFAULT_TABLE table=default.Table_254b0f36231f4cdd9d758ce8f78526c9 [id=0086b46b5e504de7a54b23886289bf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.111693 3557 consensus_queue.cc:260] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.112182 878 tablet_service.cc:1468] Processing CreateTablet for tablet e1bbc1342ffc4c0ab4508b8e6e30409f (DEFAULT_TABLE table=default.Table_254b0f36231f4cdd9d758ce8f78526c9 [id=0086b46b5e504de7a54b23886289bf21]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.112286 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1bbc1342ffc4c0ab4508b8e6e30409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.111771 3802 raft_consensus.cc:695] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:45.112498 3557 ts_tablet_manager.cc:1428] T 92dae48543b944eca15e6cbd3066c7d3 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20250905 19:52:45.113013 3557 tablet_bootstrap.cc:492] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:45.112810 3802 consensus_queue.cc:237] T c200681d4d77425389558e6489fb6032 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.114032 712 leader_election.cc:304] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 835daf30e4324782848290be181ae78f, f21c498a0d964e8fbebfd1b3cb1916fe; no voters:
I20250905 19:52:45.114394 3821 raft_consensus.cc:2802] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election won for term 1
I20250905 19:52:45.114513 3821 raft_consensus.cc:695] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 LEADER]: Becoming Leader. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Running, Role: LEADER
I20250905 19:52:45.115692 752 tablet_service.cc:1468] Processing CreateTablet for tablet b041fe456ae44521abfd92389a9cb863 (DEFAULT_TABLE table=default.Table_6c8d37e965d34470a94fab2dcc06f8a5 [id=c96c61daa43c4df4b25fea76f33a4d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.116150 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b041fe456ae44521abfd92389a9cb863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.119710 875 tablet_service.cc:1468] Processing CreateTablet for tablet b041fe456ae44521abfd92389a9cb863 (DEFAULT_TABLE table=default.Table_6c8d37e965d34470a94fab2dcc06f8a5 [id=c96c61daa43c4df4b25fea76f33a4d6b]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.111204 877 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6a10c1cfd18149b1afc933baf4654467. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.122920 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b041fe456ae44521abfd92389a9cb863. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.124790 749 tablet_service.cc:1468] Processing CreateTablet for tablet 7d6977f8d8324beda2690b8eaeb02f53 (DEFAULT_TABLE table=default.Table_79878218de474236ae52af85d653e417 [id=23ad1b84033c4ac98021b0165b6f4dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.125134 749 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d6977f8d8324beda2690b8eaeb02f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.126750 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e1bbc1342ffc4c0ab4508b8e6e30409f. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.128932 892 tablet_service.cc:1468] Processing CreateTablet for tablet 7d6977f8d8324beda2690b8eaeb02f53 (DEFAULT_TABLE table=default.Table_79878218de474236ae52af85d653e417 [id=23ad1b84033c4ac98021b0165b6f4dc9]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.129371 739 tablet_service.cc:1468] Processing CreateTablet for tablet 5fb0d2d8f2a74aafac2f98cf3e2eff34 (DEFAULT_TABLE table=default.Table_33b37ace063d463ca1b969346fc26d26 [id=77ac54f7ec034b3ea96ad14244521d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.129734 739 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb0d2d8f2a74aafac2f98cf3e2eff34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.130780 3557 tablet_bootstrap.cc:654] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
W20250905 19:52:45.135298 2291 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111) [suppressed 11 similar messages]
W20250905 19:52:45.136453 2291 consensus_peers.cc:489] T 00000000000000000000000000000000 P c401c9f3020b4ae5931c566c6bcb1711 -> Peer e38333af29db4bf6af41ee5a25d6701e (127.0.104.60:34395): Couldn't send request to peer e38333af29db4bf6af41ee5a25d6701e. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:34395: connect: Connection refused (error 111). This is attempt 11: this message will repeat every 5th retry.
I20250905 19:52:45.138825 3557 tablet_bootstrap.cc:492] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.139029 3557 ts_tablet_manager.cc:1397] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.002s
I20250905 19:52:45.140365 3557 raft_consensus.cc:357] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.140628 3557 raft_consensus.cc:383] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.140724 3557 raft_consensus.cc:738] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.141000 3557 consensus_queue.cc:260] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.142975 3557 ts_tablet_manager.cc:1428] T 1441704544e94844b0a262235318f790 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20250905 19:52:45.142980 3540 fs_manager.cc:714] Time spent opening directory manager: real 0.321s user 0.023s sys 0.022s
I20250905 19:52:45.145396 3557 tablet_bootstrap.cc:492] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:45.147753 3557 tablet_bootstrap.cc:654] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.114770 3821 consensus_queue.cc:237] T 2edf591963ca46369c29a3fb2bf24bc3 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.156311 3546 raft_consensus.cc:491] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.156451 3546 raft_consensus.cc:513] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.162439 1043 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "350197e134c549198771f4f8337d5dfc" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.162703 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "ca00bfe551a34ca9bb6d14b3a24596b6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.162883 892 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7d6977f8d8324beda2690b8eaeb02f53. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.163370 1046 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "26905425077e4746bccb51431da706c9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.167752 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "298a325f5e4645beac39eb59ddc0566d" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.168324 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe00969eb9654976a0efeac2fa7c91d3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.168473 1036 raft_consensus.cc:2466] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.168871 3557 tablet_bootstrap.cc:492] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.168859 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "97912167c32646d18f005a9039d6eca6" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.169027 3557 ts_tablet_manager.cc:1397] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.024s user 0.005s sys 0.000s
I20250905 19:52:45.169364 1045 raft_consensus.cc:2466] T 98db7e2a9d6d4f4d984388a493295d72 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 835daf30e4324782848290be181ae78f in term 1.
I20250905 19:52:45.169616 1044 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "6e3e1732bff14d49ade14fbfa6a9784c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.169750 1044 raft_consensus.cc:2466] T 6e3e1732bff14d49ade14fbfa6a9784c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.170053 3557 raft_consensus.cc:357] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.170207 875 tablet_service.cc:1468] Processing CreateTablet for tablet 734937ccc1cf4a528247cc79f893e5ae (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.170595 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734937ccc1cf4a528247cc79f893e5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.171667 752 tablet_service.cc:1468] Processing CreateTablet for tablet dff4baf04ad34a038a517e8c5142f19b (DEFAULT_TABLE table=default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.172082 752 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet dff4baf04ad34a038a517e8c5142f19b. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.173936 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5cf2bc19b043f68838f8a6af6393cd" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.170225 3557 raft_consensus.cc:383] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.176076 3557 raft_consensus.cc:738] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.176338 3557 consensus_queue.cc:260] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.177783 881 tablet_service.cc:1468] Processing CreateTablet for tablet 5fb0d2d8f2a74aafac2f98cf3e2eff34 (DEFAULT_TABLE table=default.Table_33b37ace063d463ca1b969346fc26d26 [id=77ac54f7ec034b3ea96ad14244521d83]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.178279 1169 tablet_bootstrap.cc:654] T 3c5cf2bc19b043f68838f8a6af6393cd P 4eb3af79b01d4835bf6bf7b26bf016f5: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.178010 3557 ts_tablet_manager.cc:1428] T e2659c901336484fb3e4b1a892f05792 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20250905 19:52:45.180657 881 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5fb0d2d8f2a74aafac2f98cf3e2eff34. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.185187 1036 raft_consensus.cc:1273] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: Refusing update from remote peer f21c498a0d964e8fbebfd1b3cb1916fe: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20250905 19:52:45.185279 1036 pending_rounds.cc:85] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting all ops after (but not including) 0
I20250905 19:52:45.185330 1036 pending_rounds.cc:107] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20250905 19:52:45.185420 1036 raft_consensus.cc:2887] T 51d92f92940b4952b68becf86661d459 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20250905 19:52:45.185976 3749 tablet.cc:1722] T c870e689af254fb0acb15330577e5c42 P 4eb3af79b01d4835bf6bf7b26bf016f5: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.180681 3557 tablet_bootstrap.cc:492] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
W20250905 19:52:45.184125 849 leader_election.cc:343] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.188553 849 leader_election.cc:304] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:45.189153 3681 raft_consensus.cc:2747] T cea5667d882b4b57a6fa4e9866641bf6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:45.191076 3557 tablet_bootstrap.cc:654] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.192996 3871 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20250905 19:52:45.193742 848 consensus_peers.cc:489] T 6e15350150db413fa4f100171280f46e P 835daf30e4324782848290be181ae78f -> Peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Couldn't send request to peer f21c498a0d964e8fbebfd1b3cb1916fe. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 6e15350150db413fa4f100171280f46e. This is attempt 1: this message will repeat every 5th retry.
I20250905 19:52:45.076807 3835 rpcz_store.cc:276] 0905 19:52:43.683388 (+ 0us) service_pool.cc:168] Inserting onto call queue
0905 19:52:43.683436 (+ 48us) service_pool.cc:225] Handling call
0905 19:52:45.070271 (+1386835us) inbound_call.cc:173] Queueing success response
Related trace 'op':
0905 19:52:43.683810 (+ 0us) alter_schema_op.cc:104] PREPARE ALTER-SCHEMA: Starting
0905 19:52:43.683863 (+ 53us) alter_schema_op.cc:63] Acquiring schema lock in exclusive mode
0905 19:52:43.683868 (+ 5us) alter_schema_op.cc:66] Acquired schema lock
0905 19:52:43.683872 (+ 4us) alter_schema_op.cc:119] PREPARE ALTER-SCHEMA: finished
0905 19:52:43.683891 (+ 19us) alter_schema_op.cc:128] START. Timestamp: 1757101963683725
0905 19:52:43.683893 (+ 2us) op_driver.cc:348] REPLICATION: starting
0905 19:52:43.684009 (+ 116us) log.cc:844] Serialized 173 byte log entry
0905 19:52:44.756755 (+1072746us) op_driver.cc:464] REPLICATION: finished
0905 19:52:44.860982 (+104227us) alter_schema_op.cc:133] APPLY ALTER-SCHEMA: Starting
0905 19:52:44.861291 (+ 309us) log.cc:844] Serialized 5 byte log entry
0905 19:52:45.009192 (+147901us) tablet_metadata.cc:641] Metadata flushed
0905 19:52:45.069845 (+ 60653us) log.cc:844] Serialized 14 byte log entry
0905 19:52:45.070130 (+ 285us) alter_schema_op.cc:73] Released schema lock
0905 19:52:45.070137 (+ 7us) alter_schema_op.cc:186] AlterSchemaCommitCallback: making alter schema visible
Metrics: {"child_traces":[["op",{"apply.queue_time_us":104167,"mutex_wait_us":60864,"prepare.queue_time_us":48,"prepare.run_cpu_time_us":13649,"prepare.run_wall_time_us":281511,"raft.queue_time_us":1321,"raft.run_cpu_time_us":352,"raft.run_wall_time_us":351,"replication_time_us":1072826,"thread_start_us":279900,"threads_started":1,"wal-append.queue_time_us":278341}]]}
I20250905 19:52:45.195940 741 tablet_service.cc:1468] Processing CreateTablet for tablet 734937ccc1cf4a528247cc79f893e5ae (DEFAULT_TABLE table=default.Table_a00174b61598433595c06dcf7aadbe8f [id=51ec3745c1ea49e3988def965f822e00]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.204975 741 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 734937ccc1cf4a528247cc79f893e5ae. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.207501 3546 leader_election.cc:290] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385), 835daf30e4324782848290be181ae78f (127.0.104.2:35993)
I20250905 19:52:45.209174 3557 tablet_bootstrap.cc:492] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.209331 3557 ts_tablet_manager.cc:1397] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.010s
I20250905 19:52:45.210359 3557 raft_consensus.cc:357] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.210633 3557 raft_consensus.cc:383] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.210721 3557 raft_consensus.cc:738] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.210969 3557 consensus_queue.cc:260] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.211719 3540 fs_manager.cc:730] Time spent opening block manager: real 0.067s user 0.001s sys 0.007s
I20250905 19:52:45.211745 3557 ts_tablet_manager.cc:1428] T 84a17cb780fe40e68277c89096a32aa0 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20250905 19:52:45.212067 3557 tablet_bootstrap.cc:492] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:45.212044 3540 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
uuid: "e38333af29db4bf6af41ee5a25d6701e"
format_stamp: "Formatted at 2025-09-05 19:52:11 on dist-test-slave-b1r7"
I20250905 19:52:45.213963 3557 tablet_bootstrap.cc:654] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.214087 3540 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20250905 19:52:45.216835 849 leader_election.cc:343] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.216982 849 leader_election.cc:304] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:45.217213 3791 raft_consensus.cc:2747] T 2edf591963ca46369c29a3fb2bf24bc3 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:45.220218 421 outbound_call.cc:321] RPC callback for RPC call kudu.tserver.TabletServerService.ListTablets -> {remote=127.0.104.1:34531, user_credentials={real_user=slave}} blocked reactor thread for 84813us
I20250905 19:52:45.228969 3557 tablet_bootstrap.cc:492] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.229146 3557 ts_tablet_manager.cc:1397] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20250905 19:52:45.229943 3557 raft_consensus.cc:357] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.230131 3557 raft_consensus.cc:383] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.230207 3557 raft_consensus.cc:738] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.230418 3557 consensus_queue.cc:260] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.231216 3557 ts_tablet_manager.cc:1428] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20250905 19:52:45.231498 3557 tablet_bootstrap.cc:492] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:45.233537 3557 tablet_bootstrap.cc:654] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.236730 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "fe00969eb9654976a0efeac2fa7c91d3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.236907 1036 raft_consensus.cc:3058] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.237963 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "3c5cf2bc19b043f68838f8a6af6393cd" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.238422 743 tablet_service.cc:1468] Processing CreateTablet for tablet a8ec2b670d224b5799dd2567ee8f5943 (DEFAULT_TABLE table=default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.238811 743 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ec2b670d224b5799dd2567ee8f5943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.241715 3557 tablet_bootstrap.cc:492] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.241894 3557 ts_tablet_manager.cc:1397] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.000s
I20250905 19:52:45.242877 3557 raft_consensus.cc:357] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.243130 3557 raft_consensus.cc:383] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.243220 3557 raft_consensus.cc:738] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.243476 3557 consensus_queue.cc:260] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.244201 3557 ts_tablet_manager.cc:1428] T 823e311962dc40d3b700d8d6b4a70684 P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20250905 19:52:45.244525 3557 tablet_bootstrap.cc:492] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:45.245555 744 tablet_service.cc:1468] Processing CreateTablet for tablet 9392f0fc1e774afbb524c69ef12f6a31 (DEFAULT_TABLE table=default.Table_6c3777b869db4fb5ad429e51e021b814 [id=d8cd299e01614df3a0b1c2d02b431d99]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.246321 3557 tablet_bootstrap.cc:654] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.246860 744 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9392f0fc1e774afbb524c69ef12f6a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.246990 745 tablet_service.cc:1468] Processing CreateTablet for tablet 0e71429c74b245f199fb5c8b68809b0a (DEFAULT_TABLE table=default.Table_a7fee850ff284e50b2e702f412f360ab [id=63ada09f64384488a2c03dfef40775bc]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.247942 745 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0e71429c74b245f199fb5c8b68809b0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.236403 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.251575 1045 raft_consensus.cc:2391] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:45.251719 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.251926 1040 raft_consensus.cc:2391] T 7d2e58cc18b94f9ab036e3376920351c P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:45.256865 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 414f30a772524c33a74864ff741c865f (table default.Table_e34bd3d81ce24fffaf905102775bb2e4 [id=93e1f25857f644c5bd22e445576390a7]). Expected version 1 got 0
I20250905 19:52:45.257104 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet bd9ae8fd5fdc4b539308268ba9729cd6 (table default.Table_4074e68bdbd84b8ab0734c59a600b1b7 [id=c4985b481dc94bd48f9623e4a300c498]). Expected version 1 got 0
I20250905 19:52:45.257937 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet a860ab3f5ce54bd2b75bb71730a3c6e6 (table default.Table_9bda3d1d061946baa90945381c53758b [id=10e097461e0f4f3bbf8a59753f69e913]). Expected version 1 got 0
I20250905 19:52:45.258136 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 51d92f92940b4952b68becf86661d459 (table default.Table_f055252e328e48ca9402b1265436d635 [id=848f6c88009643beb5a2abcf7d5c35b7]). Expected version 1 got 0
I20250905 19:52:45.259653 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a95130c82e14bbcae577d4244b876ab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.259913 1040 raft_consensus.cc:2466] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 0.
I20250905 19:52:45.261349 1170 tablet_bootstrap.cc:492] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe: No bootstrap required, opened a new log
I20250905 19:52:45.261570 1036 raft_consensus.cc:2466] T fe00969eb9654976a0efeac2fa7c91d3 P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:45.262037 3541 ts_tablet_manager.cc:1918] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20250905 19:52:45.262087 1170 ts_tablet_manager.cc:1397] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe: Time spent bootstrapping tablet: real 0.421s user 0.033s sys 0.044s
I20250905 19:52:45.262171 3541 log.cc:1199] T c69a41cee5ff4f8f9bccaa513c096118 P 835daf30e4324782848290be181ae78f: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-1/wal/wals/c69a41cee5ff4f8f9bccaa513c096118
I20250905 19:52:45.263002 1170 raft_consensus.cc:357] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.263227 1170 raft_consensus.cc:383] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.263302 1170 raft_consensus.cc:738] T ed8dae95b500437d869548995d84451f P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f21c498a0d964e8fbebfd1b3cb1916fe, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.265760 1036 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "4a95130c82e14bbcae577d4244b876ab" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.265834 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7a104b1c8ce14fa2bf101e1b34183cfa" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.265908 3681 raft_consensus.cc:491] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.266008 1036 raft_consensus.cc:3058] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.266053 1040 raft_consensus.cc:2391] T 7a104b1c8ce14fa2bf101e1b34183cfa P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 835daf30e4324782848290be181ae78f in this term.
I20250905 19:52:45.266036 3681 raft_consensus.cc:513] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.266482 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ca3fc7055c44df282285806ea151e51" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.266597 3681 leader_election.cc:290] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:45.267619 902 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "835daf30e4324782848290be181ae78f" is_pre_election: true
I20250905 19:52:45.267860 902 raft_consensus.cc:2391] T e476304b492f4bd69d5ea0f109263517 P 835daf30e4324782848290be181ae78f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate f21c498a0d964e8fbebfd1b3cb1916fe in current term 1: Already voted for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in this term.
I20250905 19:52:45.268268 878 tablet_service.cc:1468] Processing CreateTablet for tablet a8ec2b670d224b5799dd2567ee8f5943 (DEFAULT_TABLE table=default.Table_95826d2c4a88413cbb1afa1d780a8f0c [id=8d31dcbc2b944b7ca39b1d0246d044f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.268683 878 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a8ec2b670d224b5799dd2567ee8f5943. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.272059 764 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c4e641ac5b5c49fda12450bbf1fec2b4" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:45.272759 848 leader_election.cc:343] T c4e641ac5b5c49fda12450bbf1fec2b4 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.276068 875 tablet_service.cc:1468] Processing CreateTablet for tablet 9392f0fc1e774afbb524c69ef12f6a31 (DEFAULT_TABLE table=default.Table_6c3777b869db4fb5ad429e51e021b814 [id=d8cd299e01614df3a0b1c2d02b431d99]), partition=RANGE (key) PARTITION UNBOUNDED
W20250905 19:52:45.279022 2285 catalog_manager.cc:4726] Async tablet task AlterTable RPC for tablet 6a7e40e3ce194bbab8e460ac11282f0f (table default.Table_f864acb8d65a42d0aeced6ca942e60a8 [id=3a67ff4a6e95484a92512ecb4f1a6cff], current schema version=1) failed: Not found: failed to reset TS proxy: No leader found: Tablet 6a7e40e3ce194bbab8e460ac11282f0f consensus state has no leader
I20250905 19:52:45.281869 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736bfce682e64498a8dde3310305536a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.283756 875 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9392f0fc1e774afbb524c69ef12f6a31. 1 dirs total, 0 dirs full, 0 dirs failed
I20250905 19:52:45.286202 877 tablet_service.cc:1468] Processing CreateTablet for tablet dff4baf04ad34a038a517e8c5142f19b (DEFAULT_TABLE table=default.Table_906a087303e14db48ac1470f5c6953bd [id=577913a899e74ee8b2b1a17d7d5d27d3]), partition=RANGE (key) PARTITION UNBOUNDED
I20250905 19:52:45.288604 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "9ca3fc7055c44df282285806ea151e51" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.288559 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "736bfce682e64498a8dde3310305536a" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.290417 1045 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c200681d4d77425389558e6489fb6032" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.290843 3539 ts_tablet_manager.cc:1918] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20250905 19:52:45.291043 3539 log.cc:1199] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting WAL directory at /tmp/dist-test-taskzbeQlP/test-tmp/master-stress-test.0.CatalogManagerConfigurations_MasterStressTest.Test_0.1757101930377150-416-0/minicluster-data/ts-0/wal/wals/55cecf59d3f24db88bb10b8e12c12f82
I20250905 19:52:45.291586 3539 ts_tablet_manager.cc:1939] T 55cecf59d3f24db88bb10b8e12c12f82 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting consensus metadata
I20250905 19:52:45.292165 1049 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2edf591963ca46369c29a3fb2bf24bc3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.292408 2285 catalog_manager.cc:4928] TS f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): tablet 55cecf59d3f24db88bb10b8e12c12f82 (table default.Table_0d6fe65405f543d7a70643f94bf22da5 [id=46715848277847218d557157e4dfc77b]) successfully deleted
I20250905 19:52:45.292570 1042 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "c200681d4d77425389558e6489fb6032" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5"
I20250905 19:52:45.292838 3539 tablet_replica.cc:331] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: stopping tablet replica
I20250905 19:52:45.293113 3539 raft_consensus.cc:2241] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus shutting down.
I20250905 19:52:45.293081 1040 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "2edf591963ca46369c29a3fb2bf24bc3" candidate_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" is_pre_election: true
I20250905 19:52:45.293323 3539 raft_consensus.cc:2270] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Raft consensus is shut down!
I20250905 19:52:45.294030 3539 ts_tablet_manager.cc:1905] T 4a518797d26140f5b9abe41eab415040 P f21c498a0d964e8fbebfd1b3cb1916fe: Deleting tablet data with delete state TABLET_DATA_DELETED
W20250905 19:52:45.298887 849 leader_election.cc:343] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.299017 849 leader_election.cc:304] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:45.299208 3681 raft_consensus.cc:2747] T 350197e134c549198771f4f8337d5dfc P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:45.299559 849 leader_election.cc:343] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.299661 849 leader_election.cc:304] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:45.299840 3681 raft_consensus.cc:2747] T ca00bfe551a34ca9bb6d14b3a24596b6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:45.305015 3557 tablet_bootstrap.cc:492] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.305162 3557 ts_tablet_manager.cc:1397] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.061s user 0.012s sys 0.015s
I20250905 19:52:45.305826 3557 raft_consensus.cc:357] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.305992 3557 raft_consensus.cc:383] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20250905 19:52:45.306042 3557 raft_consensus.cc:738] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 835daf30e4324782848290be181ae78f, State: Initialized, Role: FOLLOWER
I20250905 19:52:45.306233 1036 raft_consensus.cc:2466] T 4a95130c82e14bbcae577d4244b876ab P 4eb3af79b01d4835bf6bf7b26bf016f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f21c498a0d964e8fbebfd1b3cb1916fe in term 1.
I20250905 19:52:45.306243 3557 consensus_queue.cc:260] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.306684 3557 ts_tablet_manager.cc:1428] T b23111dc8af94b56af9538f973248f2a P 835daf30e4324782848290be181ae78f: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20250905 19:52:45.307030 3557 tablet_bootstrap.cc:492] T ed02bf5743f246a1937e9a9fe22c764e P 835daf30e4324782848290be181ae78f: Bootstrap starting.
I20250905 19:52:45.308684 3557 tablet_bootstrap.cc:654] T ed02bf5743f246a1937e9a9fe22c764e P 835daf30e4324782848290be181ae78f: Neither blocks nor log segments found. Creating new log.
I20250905 19:52:45.309706 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet 4a9afd0fd4ad4ea0afcab9d7291a76bb (table default.Table_3ab8cb26205842d09aac7a486a568cd8 [id=a1026b46a81b4ea18995b4f121788776]) successfully deleted
I20250905 19:52:45.313966 3681 raft_consensus.cc:491] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.314097 3681 raft_consensus.cc:513] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.314692 3681 leader_election.cc:290] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:45.315263 768 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "30925e81025445b58dd7c2563bb48da9" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:45.315703 848 leader_election.cc:343] T 30925e81025445b58dd7c2563bb48da9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:45.320636 849 leader_election.cc:343] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.320760 849 leader_election.cc:304] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:45.320977 3681 raft_consensus.cc:2747] T 26905425077e4746bccb51431da706c9 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20250905 19:52:45.323766 849 leader_election.cc:343] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.323951 849 leader_election.cc:304] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 835daf30e4324782848290be181ae78f; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, f21c498a0d964e8fbebfd1b3cb1916fe
I20250905 19:52:45.324210 3681 raft_consensus.cc:2747] T 97912167c32646d18f005a9039d6eca6 P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20250905 19:52:45.326679 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet 26905425077e4746bccb51431da706c9 (table default.Table_092735da5fbd4ffab7939962df253fa0 [id=d66e311f2eb84a6e889e748c5ee5f7a3]). Expected version 1 got 0
I20250905 19:52:45.326969 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet f6f29ac5e68d4d8da8f3496fefe695ec (table default.Table_234e811e09aa42e2a83b43d09c7d8ee0 [id=1196f4fec5bc41109112409f35a92cb3]). Expected version 1 got 0
I20250905 19:52:45.327399 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet cea5667d882b4b57a6fa4e9866641bf6 (table default.Table_6bb624ff162648cda8ec7badc63433ed [id=68581ab2921a42d1931a9edb44fa1ecd]). Expected version 1 got 0
I20250905 19:52:45.327697 2342 catalog_manager.cc:5674] TS 835daf30e4324782848290be181ae78f (127.0.104.2:35993) does not have the latest schema for tablet af10abf0a5b34b3bbfea20e5c2005006 (table default.Table_b9646298e8214aa184d57fe2b66b6b0e [id=8213b623d8414c62874496694b1515fd]). Expected version 1 got 0
I20250905 19:52:45.327951 2291 catalog_manager.cc:4928] TS 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): tablet e58bbb6575fd4437b242032a281914ea (table default.Table_a9dc5d6bf42c4c8f852069c46921e828 [id=406ae40a142e4b6aa89bd34aa04df8e4]) successfully deleted
I20250905 19:52:45.329303 772 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "e476304b492f4bd69d5ea0f109263517" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:45.329488 772 raft_consensus.cc:3058] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.333395 757 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "1110897f9ec14741b9b882b6e09081ab" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:45.329438 770 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "7d2e58cc18b94f9ab036e3376920351c" candidate_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe"
I20250905 19:52:45.333640 757 raft_consensus.cc:3058] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.336112 3816 tablet.cc:1722] T af10abf0a5b34b3bbfea20e5c2005006 P f21c498a0d964e8fbebfd1b3cb1916fe: Alter schema from (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 0 to (
10:key INT32 NOT NULL,
PRIMARY KEY (key)
) version 1
I20250905 19:52:45.338342 711 leader_election.cc:304] T 298a325f5e4645beac39eb59ddc0566d P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:45.338495 772 raft_consensus.cc:2466] T e476304b492f4bd69d5ea0f109263517 P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:45.339603 711 leader_election.cc:343] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.348737 3641 raft_consensus.cc:491] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20250905 19:52:45.348927 3641 raft_consensus.cc:513] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35993 } }
I20250905 19:52:45.349639 3641 leader_election.cc:290] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531), 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385)
I20250905 19:52:45.352658 3546 consensus_queue.cc:1035] T 51d92f92940b4952b68becf86661d459 P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:45.352662 763 tablet_service.cc:1813] Received RequestConsensusVote() RPC: tablet_id: "32934e708cff4efa9b1189b9eecada2c" candidate_uuid: "835daf30e4324782848290be181ae78f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" is_pre_election: true
W20250905 19:52:45.354007 848 leader_election.cc:343] T 32934e708cff4efa9b1189b9eecada2c P 835daf30e4324782848290be181ae78f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer f21c498a0d964e8fbebfd1b3cb1916fe (127.0.104.1:34531): Illegal state: must be running to vote when last-logged opid is not known
I20250905 19:52:45.358357 757 raft_consensus.cc:2466] T 1110897f9ec14741b9b882b6e09081ab P f21c498a0d964e8fbebfd1b3cb1916fe [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4eb3af79b01d4835bf6bf7b26bf016f5 in term 1.
W20250905 19:52:45.365648 711 leader_election.cc:343] T 3c5cf2bc19b043f68838f8a6af6393cd P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 4eb3af79b01d4835bf6bf7b26bf016f5 (127.0.104.3:43385): Illegal state: must be running to vote when last-logged opid is not known
W20250905 19:52:45.367136 984 connection.cc:232] RPC call timeout handler was delayed by -1.0565345400000012s: this may be due to a process-wide pause such as swapping, logging-related delays, or allocator lock contention. Will allow extra 0.1s for a response
I20250905 19:52:45.367465 3801 consensus_queue.cc:1035] T 30e1ca2bd1f94f748f2022b5669bc07e P f21c498a0d964e8fbebfd1b3cb1916fe [LEADER]: Connected to new peer: Peer: permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20250905 19:52:45.368852 711 leader_election.cc:304] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: f21c498a0d964e8fbebfd1b3cb1916fe; no voters: 4eb3af79b01d4835bf6bf7b26bf016f5, 835daf30e4324782848290be181ae78f
I20250905 19:52:45.369580 3557 tablet_bootstrap.cc:492] T ed02bf5743f246a1937e9a9fe22c764e P 835daf30e4324782848290be181ae78f: No bootstrap required, opened a new log
I20250905 19:52:45.369721 3557 ts_tablet_manager.cc:1397] T ed02bf5743f246a1937e9a9fe22c764e P 835daf30e4324782848290be181ae78f: Time spent bootstrapping tablet: real 0.063s user 0.002s sys 0.007s
I20250905 19:52:45.369917 3801 raft_consensus.cc:3058] T 7a104b1c8ce14fa2bf101e1b34183cfa P f21c498a0d964e8fbebfd1b3cb1916fe [term 0 FOLLOWER]: Advancing to term 1
I20250905 19:52:45.370532 3557 raft_consensus.cc:357] T ed02bf5743f246a1937e9a9fe22c764e P 835daf30e4324782848290be181ae78f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f21c498a0d964e8fbebfd1b3cb1916fe" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34531 } } peers { permanent_uuid: "4eb3af79b01d4835bf6bf7b26bf016f5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43385 } } peers { permanent_uuid: "835daf30e4324782848290be181ae78f" member_type: VOTER last_known_addr { host: "127.0.104.2"